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SUMMARY 


Problem 


An  impressive  number  of  studies  have  been  reported  in  the  literature  comparing  the 
effects  of  various  audiovisual  media  in  presenting  academic-  or  job-related  instructional 
materials.  While  that  literature  has  reported  consistent  findings  for  certain  variables  and 
performance  settings,  the  overall  generality  of  findings  is  poor  due  to  incomplete 
specification  of  relevant  treatment  comparisons  and/or  inadequate  experimental  controls. 
The  need  for  more  systematic  research  in  this  area  is  particularly  acute  at  present  given 
rapid  advances  in  computer  technology-technology  which  has  now  made  possible  the 
combining  of  animated  graphics  and  speech-output  features  with  computer-presented 
text. 

Objectives 


The  objectives  of  this  effort  were  to  (1)  compare  the  relative  efficiency  and 
effectiveness  of  six  computer-based  methods  for  presenting  job  task  instructions, 
(2)  assess  differences  in  subjects'  performance  based  on  their  prior  training/experience  in 
performing  the  tasks  sampled,  (3)  determine  any  interaction  effects  between  the 
foregoing  factors,  and  (4)  test  certain  predictions  suggested  by  Paivio's  dual-coding  theory 
of  information  processing  in  an  applied  setting.  The  theory  posits  that  information 
represented  by  both  verbal  and  image  codes  is  more  powerful  than  that  represented  by 
either  coding  system  alone. 

Method 


Two  subject  groups  were  formed:  one  consisted  of  90  Navy  enlisted  personnel  who  had 
prior  training  and  experience  in  operating  oscilloscopes;  the  other,  of  90  Navy  recruits 
who  had  no  such  prior  training  or  experience.  Subjects  in  each  group  were  randomly 
assigned  to  one  of  six  instructional  presentation  conditions:  text-only,  audio-only,  text- 
audio,  text-graphics,  audio-graphics,  and  text-audio-graphics,  resulting  in  15  subjects  per 
condition.  Each  subject  performed  the  following  oscilloscope  operator  tasks  sequentially: 
(1)  initial  setup  and  adjustments,  (2)  probe  calibration,  (3)  amplitude  measurement,  and 
(4)  frequency  measurement.  Five  performance  measures  were  obtained  for  each  task:  time- 
to-completion,  passing  or  failing  performance  scores,  and  separate  frequency  counts  for 
task  steps  that  were  repeated,  performed  incorrectly,  or  performed  out  of  sequence. 

A  2  x  6  between-subjects,  multiple  analysis  of  variance  (MANOVA)  served  as  the 
model  for  testing  the  following  experimental  hypotheses:  (1)  trained  and  experienced 
oscilloscope  operators  will  demonstrate  better  task  performance  than  untrained,  inexperi¬ 
enced  ones;  (2)  for  both  groups,  subjects  assigned  to  presentation  methods  that  include 
graphics  will  demonstrate  better  task  performance  than  subjects  assigned  to  methods  that 
exclude  graphics  (3)  for  both  groups,  no  differences  in  task  performance  will  be  found 
between  subjects  assigned  to  the  text-audio  method  and  subjects  assigned  to  the  text-only 
and  audio-only  methods;  (4)  for  both  groups,  no  difference  in  task  performance  will  be 
found  between  subjects  assigned  to  the  text-audio-graphics  method  and  subjects  assigned 
to  the  text-graphics  and  audio-graphics  methods. 

Results 


Significant  main  effects  were  obtained  for  the  experience  and  presentation  method 
variables,  but  the  interaction  between  them  was  nonsignificant.  As  predicted,  the 
performance  of  the  experienced  subjects  was  significantly  better  than  that  of  the 
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inexperienced  ones,  with  the  most  pronounced  differences  occurring  on  the  probe  and 
amplitude  tasks.  Performance  of  subjects  using  methods  that  included  graphics  informa¬ 
tion  was  significantly  better  than  that  of  subjects  whose  methods  excluded  graphics  for  all 
but  the  frequency  task.  These  findings  were  interpreted  as  providing  substantial  support 
for  the  second  experimental  hypothesis.  Contrary  to  the  third  and  fourth  hypotheses, 
subjects  assigned  to  the  text-only  and  text-graphics  methods  repeated  a  significantly 
higher  number  of  steps  on  all  tasks  and  had  significantly  higher  failure  rates  on  certain 
tasks  than  subjects  whose  methods  included  audio  instructions. 

Conclusions  and  Implications 

The  data  suggest  that  more  efficient  and  effective  performance  may  result  when 
highly  proceduralized  job  task  instructions  are  presented  in  a  combined  audio-graphics 
format.  This  effect  can  be  enhanced  further  through  the  inclusion  of  redundant  task 
instructions  in  textual  form.  It  is  important  to  note,  however,  that  the  foregoing  methods 
had  their  greatest  impact  early  in  the  performance  of  consecutively  administered  job  task 
procedures;  that  is,  when  all  subjects  were  becoming  familiar  with  the  location  and 
function  of  particular  oscilloscope  controls.  Moreover,  the  results  lend  support  to  Paivio's 
dual-coding  theory  of  information  processing  in  an  applied  setting. 
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INTRODUCTION 


Problem  and  Background 

People  in  many  sectors  of  working  life  increasingly  depend  on  sophisticated  equip¬ 
ment  and  detailed  procedures  to  assist  them  in  performing  their  jobs.  The  users  of  such 
equipment  and  procedures  must  often  rely  on  information  of  a  technical  nature,  ranging 
from  simple,  step-by-step  instructions  to  complex  concepts  involving  system  interrela¬ 
tionships  and  theory  of  operation.  In  this  connection,  the  writers  of  job-related  technical 
information  are  continually  faced  with  the  problem  of  selecting  among  various  methods 
the  best  one  for  organizing,  formatting,  and  ultimately  presenting  this  type  of  information 
to  users.  All  too  often,  choices  are  made  from  among  available  methods,  with  little  or  no 
knowledge  of  the  comprehensibility  or  general  usefulness  of  the  resultant  information, 
particularly  from  the  user's  perspective. 

One  method  for  dealing  effectively  with  problems  of  organizing  and  formatting  job- 
related  technical  information  has  involved  the  development  of  prescriptive  manuals, 
commonly  referred  to  as  job  performance  aids  (JPAs).  A  primary  focus  of  this  method 
involves  modeling  the  job  activities  the  user  is  expected  to  perform,  then  using  the  model 
to  predict  the  user's  information  needs  and  how  the  technical  information  should  be 
organized  to  meet  those  needs.  The  JPA  method  also  takes  the  user's  cognitive  skills  into 
consideration  through  the  use  of  specific  guidelines  that  prescribe  the  formatting  of  text 
and  instructions.  Besides  containing  standard  verb  lists  for  describing  job  task  steps  in 
discrete,  action-oriented  terms,  these  guidelines  provide  rules  for  formatting  task  steps  in 
a  checklist  fashion.  Some  rules  limit  the  number  of  words,  sentences,  and  steps  per  page 
to  facilitate  rapid  scanning  and  retention  of  the  information.  Other  rules  often  specify 
the  pairing  of  task  steps  with  drawings  that  show  the  location  of  parts,  and/or  the  physical 
actions  needed  to  perform  each  step.  An  example  of  a  typical  JPA  is  given  in  Figure  1. 

Research  on  JPAs  conducted  since  the  early  1950's  has  been  reviewed  extensively  by 
Booher  (1978),  Foley  and  Camm  (1972),  Rowan  (1973),  and  Smillie  (1985).  The  major 
conclusions  that  can  be  drawn  from  these  reviews  are  that  the  use  of  JPAs  not  only 
improves  the  clarity  of  the  technical  information  needed  to  perform  job  tasks,  but  also 
reduces  the  requirement  for  extensive  training  and  experience  of  the  user.  For  example, 
Foley  and  Camm  (1972)  reported  that  inexperienced  Navy  maintenance  technicians  who 
used  JPAs  performed  the  same  job  tasks  as  well  as  more  experienced  technicians  who  did 
them  routinely,  with  or  without  the  use  of  conventional  technical  manuals.  Even  highly 
trained  and  experienced  job  incumbents  made  fewer  errors  when  using  JPAs  under 
controlled  test  conditions  (Foley,  1973). 

While  the  findings  from  these  and  other  studies  have  shown  JPAs  to  be  effective  in 
improving  job  task  performance,  several  problems  have  inhibited  their  widespread  use  in 
civilian  and  military  work  settings.  For  one  thing,  since  JPAs  are  intended  primarily  as 
supplements  to  conventional  technical  manuals,  they  add  to  the  steady  proliferation  of 
technical  information  (Duffy,  1985;  Palmer  &  Stembler,  1983;  Sulit  &  Fuller,  1976).  For 
another,  the  development  of  JPAs  frequently  requires  substantial  investment  both  in 
terms  of  front-end  analyses  and  subsequent  verification  of  the  information.  These  factors 
result  in  JPAs  that  are  equally  as,  and  in  some  cases  more,  costly  to  design,  produce,  and 
update  as  conventional  technical  manuals. 
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REMOVE  RUDDER  CONTROL  PRESSURE  SWITCH 


6-5 


Install  rudder  lock. 


1.  Request  that  assistant  hold  rudder 
in  faired  neutral  position. 


2.  Remove  left  bolt. 


3,  Place  lock  assembly  around  torque 
tube  from  left  aide.  Engage  lock 
pins  through  forward  and  aft  holes 
of  upper  flange. 


4.  Lower  and  engage  center  lock  pin 
through  lower  flange  left  bolt  hole. 


5.  Request  that  rudder  be  released. 


6.  Place  streamer  outside  of  aircraft 
through  open  tail  cone  or  tail  cone 
access  door. 
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Figure  1. 


An  example  of  a  JPA  (from  Goff,  Schlesinger,  &  Parlog,  1969). 


Recognizing  these  problems  and  tradeoffs,  civilian  and  military  planners  have 
devoted  considerable  attention  in  recent  years  to  the  development  and  use  of  new 
methods  for  presenting  job-related  technical  information,  especially  the  use  of  portable 
"stand-alone"  microcomputers.  In  addition  to  the  potential  that  such  devices  offer  for 
storing  and  providing  rapid  access  to  large  amounts  of  technical  information,  recent 
advances  in  computer  technology  have  made  possible  the  inclusion  of  a  variety  of 
enhancements,  such  as  animated  graphics  and  text-to-speech  output  features.  To  date, 
however,  there  is  a  paucity  of  research  in  the  literature  demonstrating  benefits  to  the 
user  of  automating  the  delivery  of  technical  information.  Moreover,  there  have  been  no 
definitive  comparisons  reported  in  the  literature  regarding  current  options  for  presenting 
job-related  technical  information  via  computer. 

Objectives 

The  present  study  was  designed  to  (1)  compare  the  relative  efficiency  and  effective¬ 
ness  of  six  alternative  computer-based  methods  for  presenting  job  task  instructions, 
(2)  assess  differences  in  subjects'  performance  based  on  their  prior  task  training  and 
experience,  (3)  determine  any  interaction  effects  between  the  foregoing  factors,  and 
(4)  test  certain  predictions  of  a  dual-coding  theory  of  information  processing  in  an  applied 
research  setting. 


REVIEW  OF  THE  LITERATURE 

A  detailed  literature  review  was  conducted  to  gain  a  better  understanding  of  the 
effects  of  audio  and  visual  presentations  on  human  learning  and  performance.  The  basic 
source  materials  used  in  the  literature  survey  included  standard  psychological,  education¬ 
al,  and  audiovisual  communications  journals;  technical  reports  from  civilian  and  military 
research  centers;  and  bibliographic  searches  of  computerized  data  banks  of  the  National 
Technical  Information  Service  (NTIS)  and  Educational  Research  Information  Center 
(ERIC).  This  search  was  aided  greatly  by  several  reviews  and  annotated  bibliographies  of 
the  audiovisual  communications  literature,  including  those  of  Chalupsky  and  Kopf  (1967), 
Day  and  Beach  (1950),  Hartman  (1961b),  Levie  and  Dickie  (1973),  and  Sticht,  Beck,  Hauke, 
Kleiman,  and  Oames  (1974). 

The  literature  review  focused  on  studies  that  compared  the  effectiveness  of  audio 
and  visual  media  in  presenting  academic  or  job-related  instructional  materials.  Theories 
and  models  concerned  with  human  communication  and  information  processing  were 
considered  only  insofar  as  they  provided  a  basis  for  understanding,  explaining,  or 
measuring  the  effects  of  various  audiovisual  presentations  from  both  basic  and  applied 
research  perspectives.  The  publications  were  divided  into  the  following  content  areas: 
(1)  studies  relevant  to  academic  instructional  materials  and  settings,  (2)  studies  concerned 
with  job-related  instructional  materials  in  applied  settings,  and  (3)  publications  dealing 
with  model  or  theory  development.  The  studies  in  the  first  two  content  areas  were 
subdivided  further  to  specify  the  number  and  type  of  audiovisual  presentation  methods 
compared,  while  the  third  content  area  classified  publications  within  the  framework  of 
two  theoretical  perspectives. 

Studies  in  Academic  Settings 

For  the  most  part,  studies  in  this  area  were  restricted  to  those  that  compared  the 
effects  of  various  audio  and  visual  presentations  on  the  learning  of  meaningful,  connected 
discourse  (e.g.,  sentences,  passages,  or  lecture-type  materials),  and  those  that  used  young 
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adults,  ranging  from  high  school  students  to  military  enlistees,  as  test  subjects.  The 
former  limitation  was  imposed  primarily  because  of  the  close  relationship  that  such 
materials  have  to  traditional  methods  for  presenting  job-related  information;  the  latter, 
to  provide  a  basis  for  extending  methodologies  and/or  generalizing  the  findings  from  these 
studies  to  the  present  research. 

Comparisons  of  Single-channel  Presentations 

The  literature  presents  some  conflicting  results  regarding  the  use  of  text-only,  audio- 
only,  or  pictures-only  as  separate  channels  for  communicating  instructional  materials. 
Levie  and  Dickie  (1973),  for  example,  cite  six  studies  that  showed  that  reading  was 
superior  to  listening  for  high  school  and  college  students  when  learning  was  tested  either 
by  immediate  or  by  longer-term  recall.  Similar  findings  were  obtained  in  nine  studies 
reviewed  by  Hartman  (1961b),  in  which  text  was  found  to  be  superior  to  audio  for  the 
retention  of  meaningful  prose  among  adult  subjects. 

Contrary  results  for  the  same  channel  comparisons  involving  the  retention  of  prose 
materials  among  college  students  were  reported  by  Worcester  (1925),  Greene  (1934),  and 
Goldstein  (1940).  Goldstein's  study  was  the  only  one  reviewed  in  which  reading  rate  was 
paced  line-by-line  to  match  the  rate  of  presenting  the  same  materials  aurally.  With  this 
control  of  reading  rate,  listening  comprehension  exceeded  reading  comprehension. 

Several  publications  reported  no  reliable  differences  between  text  and  audio  present¬ 
ations.  These  included  studies  that  dealt  with  college-level  prose  material  (Nasser  & 
McEwen,  1976);  remedial  mathematics  instruction  for  enlisted  Navy  personnel  (Main, 
1974);  and  a  set  of  studies  reviewed  by  Sticht,  Beck,  Hauke,  Kieiman,  and  James  (1974)  in 
which  listening  and  reading  comprehension  skills  were  generally  found  to  be  equivalent. 
Similarly,  near  equivalence  in  reading  and  listening  comprehension  skills  was  confirmed  in 
subsequent  research  by  Sticht,  Hooke,  and  Caylor  (1982)  for  young  adults  whose  reading 
abilities  varied  from  the  2nd  through  11th  grade  levels.  These  findings  were  based  on  data 
from  more  than  2,000  prospective  enlistees  for  military  service. 

Conflicting  findings  have  also  been  reported  in  the  literature  when  comparing 
learning  from  pictures  with  that  from  text  or  audio  presentations.  Of  14  studies  cited  by 
Hartman  (1961b),  5  found  an  advantage  for  the  pictorial  channel  over  the  text;  6  found  an 
advantage  for  the  pictorial  channel  over  audio;  and  3  found  no  differences.  More 
recently,  Baggett  (1979)  compared  college  students'  recall  of  a  story  in  text  and  in  a  silent 
film  version  and  found  no  difference  between  these  presentation  modes.  Gropper  (1966) 
and  Levin  (1973)  also  found  no  significant  differences  between  learning  from  pictorial  and 
textual  materials. 

Comparisons  of  Dual-channel  Presentations 


These  studies  dealt  with  comparisons  of  dual  channel  presentations  with  one  another 
or  with  single  channel  presentations.  For  this  review,  dual  channel  was  defined  as  the 
simultaneous  presentation  of  the  same  material  by  combining  two  separate  information 
channels  (e.g.,  text  and  audio,  text  and  pictures,  audio  and  pictures).  As  noted  by 
Hartman  (1961a),  there  are  four  possible  ways  information  can  be  presented  using  two 
channels:  it  can  be  redundant,  related,  unrelated,  or  contradictory.  Since  the  most 
common  situations  encountered  in  educational  settings  are  those  involving  the  simultane¬ 
ous  presentation  of  redundant  or  related  information,  studies  involving  the  other  varia¬ 
tions  were  not  considered. 
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In  a  review  by  Day  and  Beach  (1950),  nine  studies  were  cited  in  which  a  combined 
text  and  audio  presentation  of  material  led  to  more  efficient  comprehension  than  the 
presentation  of  either  text  or  audio  alone.  Webb  and  Wallon  (1956)  and  Hartman  (1961a) 
concluded  from  similar  results  that  a  simultaneous  text-audio  presentation  is  more 
effective  than  either  text  or  audio  alone,  provided  that  the  information  presented  in  the 
two  channels  is  redundant. 

Exceptions  to  the  foregoing  have  been  reported  in  other  studies,  however.  Nasser  and 
McEwen  (1976)  found  that  a  combined  text-audio  format  was  superior  to  audio  alone,  but 
that  the  dual  presentation  had  no  advantage  over  a  text-only  format  when  dealing  with 
college-level  prose  material.  Main  (1974)  reported  that  supplementing  printed  texts  with 
verbatim  audio  recordings  failed  to  improve  the  effectiveness  of  a  Navy  remedial 
mathematics  course.  A  possible  reason  offered  for  the  failure  of  the  audio  supplement  to 
improve  test  performance  in  this  study  was  that  the  printed  materials  intentionally 
provided  instruction  with  low  verbal  content. 

Thus,  equivocal  findings  have  been  reported  for  comparisons  between  combined  text 
and  audio  channels  and  single  channel  presentations.  However,  numerous  studies  have 
demonstrated  that  the  addition  of  pictures  to  text  or  audio  generally  increases  learning. 
Moreover,  the  positive  effect  of  pictures  has  been  empirically  validated  across  a  diverse 
set  of  subject  groups  and  instructional  materials.  Rasco,  Tennyson,  and  Boutwell  (1975) 
found  that  pictorial  additions  facilitated  high  school  students'  comprehension  of 
mathematical  concepts.  Dwyer  (1967,  1968)  discovered  that  the  addition  of  pictures 
improved  the  effectiveness  of  physiology  instruction,  whether  it  was  presented  to  high 
school  students  in  texts  or  to  college  students  in  lectures.  Sellman  (1972)  reported  that 
the  effectiveness  of  an  Air  Force  text  on  fire  fighting  was  improved  by  adding  pictorial 
presentations.  Still  other  studies  have  shown  that  pictures  can  serve  as  advance 
organizers  for  instructional  materials  (e.g.,  Bransford  &  Johnson,  1972;  Brody  &  Legenza, 
1980;  Snowman  &  Cunningham,  1975). 

In  comparing  two-channel  presentations  with  one  another,  several  studies  have  found 
an  advantage  for  pictorial  over  printed  augmentation  of  orally  presented  information 
(Peng  <5c  Levin,  1979;  Rowher  <5c  Harris,  1975;  Rowher  &  Matz,  1975).  It  should  be  noted, 
however,  that  these  findings  were  based  on  relatively  simple  prose  materials  administered 
to  elementary  school  children.  Similar  comparisons  involving  more  complex  instructional 
materials  administered  to  Navy  trainees  (Main  &  Griffiths,  1977)  found  no  difference 
between  combined  text-pictorial  and  audio-pictorial  presentations. 

Comparisons  of  Multiple-channel  Presentations 

The  three  studies  reviewed  in  this  subcategory  compared  the  combination  of  text, 
audio,  and  pictorial  information  channels  with  the  same  three  channels  used  separately  or 
in  various  pairs.  In  a  study  by  Hartman  (1961a),  groups  of  college  students  were  given  a 
perceptual  discrimination  task  using  one  of  the  foregoing  presentation  conditions.  His 
results  showed  the  text-audio  combination  to  be  the  most  consistently  effective  of  the 
seven  conditions  tested.  Studies  conducted  by  Nugent1  (1982)  and  Rowher  and  Harris 
(1975)  also  compared  these  same  treatment  conditions  using  elementary  school  children  as 
subjects.  Unlike  Hartman  (1961a),  these  authors  found  no  advantage  for  the  combined 
text  and  audio  condition  over  any  of  the  single-channel  presentations.  Their  results  did, 
however,  indicate  that  the  combination  of  pictures  with  text  or  audio  generally  maxi¬ 
mized  learning.  A  consistent  finding  in  all  three  studies  was  that  performance  in  the 
combined  text,  audio,  and  pictorial  condition  was  no  better  than  that  in  various  dual¬ 
channel  presentations. 

1  No  relation  to  the  author  of  this  report. 
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Studies  in  Applied  Settings 


The  studies  reviewed  in  this  content  area  compared  the  effects  of  various  audio  and 
visual  presentation  methods  in  providing  the  information  needed  to  perform  actual  or 
simulated  job  tasks.  These  studies  also  included  a  diverse  set  of  subjects,  ranging  from 
college  students  to  factory  workers. 

Comparisons  of  Single-channel  Presentations 

The  two  studies  included  in  this  area  examined  the  effects  of  presenting  instructional 
materials  in  text  and  pictorial  form.  In  one  study,  Booher  (1975)  compared  the  speed  and 
accuracy  with  which  Navy  enlisted  personnel  performed  three  different  types  of  simulated 
job  tasks  using  the  above  presentation  methods.  On  the  average,  subjects  in  the  pictorial- 
only  condition  completed  all  tasks  in  about  one-third  less  time  than  subjects  in  the  text- 
only  condition.  This  significant  increase  in  speed  was  at  the  expense  of  accuracy, 
however,  in  that  subjects  assigned  to  the  text-only  condition  made  significantly  fewer 
errors  than  subjects  in  the  pictorial-only  condition. 

Contrary  findings  on  the  accuracy  of  task  performance  using  these  same  two 
presentation  methods  were  reported  by  Stone  and  Glock  (1981).  In  this  study  groups  of 
college  students  received  instructions  for  assembling  a  scale-model  loading  cart.  Results 
showed  no  difference  in  the  total  number  of  errors  made  between  groups  of  subjects  who 
viewed  only  text  and  those  who  viewed  only  illustrations.  When  analyses  were  performed 
on  specific  types  of  errors,  however,  it  was  found  that  subjects  in  the  pictorial-only 
condition  made  significantly  fewer  errors  of  orientation  (i.e.,  parts  of  the  model  attached 
at  the  proper  place  but  not  properly  oriented  in  space)  than  subjects  in  the  text-only 
condition.  Time  measures  for  task  performance  were  not  obtained  in  this  study. 

Comparisons  of  Dual-channel  Presentations 

The  foregoing  studies  also  included  comparisons  of  various  dual-channel  presentations 
involving  the  combination  of  text  and  pictorial  instructions.  The  Stone  and  Glock  (1981) 
study  showed  that  subjects  in  the  combined  text-pictorial  condition  made  significantly 
fewer  errors  in  the  assembly  task  than  subjects  in  either  the  text-only  or  pictorial-only 
condition.  Further,  subjects  in  the  text-pictorial  condition  made  significantly  fewer 
errors  in  orientation  of  parts  than  text-only  subjects,  but  no  fewer  errors  than  pictorial- 
only  subjects. 

In  addition  to  the  single-channel  presentation  methods  described  above,  Booher's 
(1975)  study  also  included  four  dual-channel  conditions.  Two  of  these  consisted  of 
predominantly  textual  formats,  supplemented  with  either  related  or  redundant  pictorial 
information;  and  two  consisted  of  predominantly  pictorial  formats,  supplemented  with 
either  related  or  redundant  text.  Booher  found  no  differences  among  the  pictorial-only 
and  the  predominantly  pictorial  formats  with  respect  to  overall  speed  and  accuracy  of 
task  performance;  however,  subjects  in  these  three  conditions  were  faster  and  made 
significantly  fewer  errors  than  subjects  in  the  other  conditions.  In  addition,  his  results 
showed  that  subjects  in  the  text-only  condition  took  significantly  longer  to  complete  all 
tasks  than  subjects  who  used  the  other  presentation  formats.  Booher  concluded  that  the 
predominantly  pictorial  formats  were  the  only  ones  that  resulted  in  consistently  fast  and 
accurate  performance. 

The  final  study  in  this  subcategory  (Goff,  Schlesinger,  &  Parlog,  1969)  compared  the 
effects  of  presenting  job  task  instructions  when  using  a  combined  text-pictorial  condition 
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with  those  when  using  a  combined  audio-pictorial  condition.  The  test  subjects  were  36  Air 
Force  enlisted  personnel,  half  of  whom  had  not  received  any  training  or  experience  in  the 
maintenance  of  the  C-141  jet  aircraft  used  in  the  study.  The  other  half  consisted  of 
trained  personnel  who  had  approximately  2  years  of  relevant,  on-job  maintenance 
experience.  Step-by-step  instructions  for  performing  various  C-141  aircraft  maintenance 
tasks  either  were  presented  in  booklet  form  or  were  narrated  to  the  subjects  by  means  of 
an  audio  tape.  The  text  and  audio  instructions  were  supplemented  with  outline  drawings 
that  showed  the  location  of  parts  and/or  the  actions  required  to  perform  the  task  steps. 
(An  example  of  instructions  presented  in  the  text-pictorial  condition  is  provided  in  Figure  T.) 

There  was  no  difference  between  the  two  groups  in  terms  of  the  number  of  errors 
made.  However,  significant  differences  in  task  completion  times  were  obtained  for 
experience  level,  presentation  method,  and  for  the  interaction  of  these  factors. 
Specifically,  results  showed  that:  (1)  inexperienced  subjects  took  significantly  longer  to 
complete  component  removal  and  replacement  tasks  than  experienced  ones,  and  (2) 
subjects  who  received  instructions  in  booklet  form  were  faster  than  those  who  received 
them  via  audio  tape,  both  in  completing  component  removal  tasks  and,  for  experienced 
subjects,  in  completing  all  tasks. 

Comparisons  of  Multiple-channel  Presentations 

The  studies  in  this  subcategory  compared  the  effectiveness  of  various  audiovisual 
devices  in  presenting  instructions  for  equipment-operator,  maintenance,  and  assembly 
tasks  performed  by  civilian  technicians  and  factory  workers.  A  study  by  Brown  ( 1964) 
involved  comparisons  of  two  audiovisual  devices  that  provided  the  information  needed  to 
align  the  master  indicator  console  on  a  military  radar.  One  device  presented  integrated 
text,  audio,  and  pictorial  instructions  by  means  of  a  lightweight  apparatus  worn  on  the 
subject's  head;  the  other  provided  identical  information  on  a  much  larger,  portable  unit. 
Equal  numbers  of  experienced  and  inexperienced  technicians  performed  the  alignment 
twice,  using  the  head  device  for  one  session  and  the  portable  device  for  the  other. 

Results  showed  that  the  significant  differences  in  the  time  required  to  perform  the 
alignment  were  a  function  of  experience  and  trial.  Specifically,  the  inexperienced 
technicians  took  longer  to  complete  both  tasks  than  than  the  experienced  ones,  while  the 
second  alignment  procedure  was  performed  significantly  faster  than  the  first,  regardless 
of  the  device  used  to  present  the  information.  No  differences  were  found  in  the  time 
measure  as  a  function  of  presentation  method. 

Chalupsky  and  Kopf  (1967)  presented  the  results  of  interviews  with  representatives 
from  12  companies  in  private  industry  who  had  experience  with  the  use  of  audiovisual 
JPAs.  Included  in  this  publication  was  an  annotated  bibliography  of  relevant  research  in 
the  area.  Items  2,  11,  21,  and  23  from  their  review  described  benefits  associated  with 
using  the  same  portable  unit  tested  in  Brown's  (1  964)  study.  In  general,  these  studies 
reported  that  the  audiovisual  device  significantly  increased  worker  productivity  and 
quality  of  finished  products,  and  significantly  reduced  the  need  for  extensive  entry-level 
job  training  and  close  supervision. 

Theories  and  Models 


The  publications  reviewed  in  this  section  stem  largely  from  two  major  perspectives: 
communication  theory  and  information  processing  theory.  These  perspectives  differ  in 
terms  of  their  focus  and  objectives,  but  each  arose  from  a  need  to  understand  better  the 
complexities  of  human  communication  and  information  processing. 
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Communication  Theory 


Probably  the  most  important  model  developed  in  this  area  was  that  of  Shannon  and 
Weaver  (1949).  Their  work  involved  a  mathematical  theory  of  transmission  of  electronic 
signals  and  described  the  circumstances  under  which  perfect  transmission  would  occur  in 
the  presence  of  "noise."  Part  of  their  explanation  took  the  form  of  an  engineering  model 
that  contained  the  following  elements:  (1)  a  source  of  information,  (2)  an  encoder,  (3)  a 
channel  for  transmission  of  signals,  (4)  a  source  of  noise,  (5)  a  decoder,  and  (6)  a  message 
destination — all  arranged  in  linear  order. 

An  implicit  assumption  of  this  model  is  that  the  same  alphabet  (i.e.,  any  set  of  signs 
such  as  letters,  dots,  dashes,  or  numbers)  exists  at  both  the  transmitting  and  receiving 
ends  of  the  communication  channel.  Thus,  according  to  the  model,  it  is  the  function  of 
the  information  source  to  select  signs  from  this  alphabet  to  constitute  a  message.  The 
message  is  then  transmitted  in  physical  form  as  a  signal,  through  a  communication 
channel,  to  the  decoder.  At  the  decoder,  the  signal  then  operates  upon  an  identical 
alphabet  that  selects  corresponding  signs  for  reconstructing  the  message  before  it  reaches 
the  destination.  Further,  it  is  assumed  that  the  stimulation  available  to  the  receiver  is 
message  plus  noise.  The  latter  includes  not  only  noise  in  the  literal  sense  of  the  term,  but 
also  irrelevancies  intermingled  with  the  intended  message. 

The  Shannon-Weaver  model  included  mathematical  expressions  to  quantify  measures 
of  various  aspects  of  communication  efficiency,  dependability,  redundancy,  and  channel 
capacity.  This  ability  to  obtain  quantitative  measurements  in  the  area  of  communication 
aroused  interest  in  many  fields  of  research  and  numerous  applications  followed  (e.g., 
Attneave,  1959;  Cherry,  1957;  Hsia,  1968a,  1968c).  Beyond  its  technical  use,  the  model 
facilitated  the  discussion  of  different  approaches  to  communication  processes  by  provid¬ 
ing  a  common  base.  These  positive  attributes  notwithstanding,  Travers  (1970)  criticized 
the  model  as  being  too  "mechanistic"  by  noting  that  it  placed  a  premium  on  the  physical 
elements  of  communication  and  ignored  syntactics,  semantics,  and  the  practical  elements 
of  communication.  In  the  same  vein,  Cherry  (1957,  p.  170)  noted  that  the  "theory  is 
written  in  the  meta-language  of  an  external  observer;  it  is  not  a  description  of  the  process 
of  communication  as  it  appears  to  one  of  the  participants." 

Information  Processing  Theory 

As  disillusionment  with  communication  theory  spread,  researchers  began  developing 
alternative  theoretical  formulations,  collectively  referred  to  as  information  processing 
theory,  to  guide  their  work.  The  primary  goal  of  this  theory  is  to  explain  the  properties  of 
mental  behaviors  and  processes  responsible  for  receiving,  storing,  retrieving,  and  using 
information  that  may  originate  either  in  the  external  environment  or  from  internal  mental 
states. 

An  important  and  influential  construct  in  this  area  was  the  limited-capacity  channel 
proposed  by  Broadbent  (1958).  Part  of  Broadbent's  theory  took  the  form  of  an  explanatory 
model  whose  predictions  could  be  quantified;  the  model  itself  did  not,  however,  purport  to 
be  a  mathematical  model.  The  core  concept  of  this  model  is  a  limited-capacity  channel, 
identified  as  the  "P-system,"  that  can  handle  only  a  given  amount  of  information  in  a 
given  time.  Broadbent  (1965)  likened  this  limited-capacity  system  to  the  flow  of  cars  on  a 
freeway:  Each  lane  can  handle  only  so  many  vehicles  per  hour;  thus  its  limitation  is  based 
on  the  rate  at  which  "traffic"  may  flow  rather  than  on  the  total  number  of  "vehicles"  that 
can  be  accommodated. 
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The  model  also  stipulates  the  conditions  under  which  the  P-system  acts  as  a  limited- 
capacity  channel.  When  the  rate  of  incoming  information  from  two  or  more  perceptual 
systems  is  sufficiently  low,  all  of  the  information  may  pass  directly  to  the  P-system.  At 
faster  rates  of  presentation,  however,  the  model  predicts  that  the  P-system  eventually 
reaches  a  point  at  which  it  can  no  longer  handle  simultaneous  inputs  from  multiple 
sources.  Instead,  it  will  accommodate  information  from  only  a  single  source,  thereby 
functioning  as  a  sequential,  single-channel  utilization  system. 

In  addition  to  encompassing  findings  from  his  own  research  efforts,  Broadbent's  (1958) 
model  provided  a  theoretical  framework  that  allowed  collation  of  a  substantial  body  of 
empirical  findings  from  many  fields  of  research.  For  example,  the  work  of  Travers  (1964, 
1967,  1970)  and  his  associates  (e.g.,  Oester  &  Travers,  1966;  Reid  &  Travers,  1968;  Van 
Mondfrans  &  Travers,  1964)  extended  Broadbent's  limited-capacity  model  to  the  area  of 
audiovisual  communications.  These  studies  generally  showed  that  combined  text-audio 
presentations  of  redundant  information  offered  no  advantage  over  either  of  these  two 
channels  alone,  even  when  the  rate  of  presenting  information  was  relatively  slow  (e.g., 
200  words  per  minute  or  less). 

The  Travers  et  al.  studies  showed  that,  at  faster  rates  of  presentation,  two  things 
began  to  happen.  First,  evidence  of  P-system  "jamming"  was  obtained  for  the  combined 
text-audio  condition,  suggesting  that  the  use  of  two  information  channels  resulted  in 
interference  of  one  channel  with  the  other.  And  second,  as  this  point  of  information 
jamming  was  reached,  subjects  took  steps  to  block  out  the  nonpreferred  information 
channel  (e.g.,  by  closing  their  eyes  and  listening  intently,  or  by  putting  their  hands  over 
their  ears  to  block  the  sound,  then  reading  the  information).  Collectively,  these  findings 
were  interpreted  as  providing  support  for  a  basic  postulate  of  the  limited-capacity  model: 
that  information  from  two  or  more  sources  must  ultimately  pass  through  a  sequential,  as 
opposed  to  a  simultaneous,  information  processing  system. 

Despite  the  major  contribution  made  by  Broadbent's  model  to  information  processing 
theory,  various  aspects  of  the  model  were  criticized  as  additional  research  began  to 
reveal  inconsistencies.  With  respect  to  the  limited-capacity  channel,  Mowbray  and 
Rhodes  (1959)  found  that,  after  sufficient  practice,  reaction  time  for  selecting  among 
eight  simultaneously  presented  response  alternatives  was  as  low  as  that  for  two  choices. 
Other  studies  showed  that,  in  situations  having  high  stimulus-response  compatibility, 
reaction  time  did  not  depend  on  the  number  of  response  alternatives  (e.g.,  Davis,  Moray, 
&  Treisman,  1961;  Leonard,  1959).  An  additional  criticism  of  the  Broadbent  model  is  that 
it  focuses  almost  exclusively  on  differences  in  learning  or  performance  attributable  to 
such  factors  as  channel  capacity  and  the  rate  at  which  information  is  presented.  As  a 
result,  it  fails  to  consider  any  differential  advantages  afforded  by  the  auditory  and  visual 
sensory  modalities— either  separately  or  combined— as  mechanisms  for  information  pro¬ 
cessing.  These  shortcomings  were  addressed  by  the  dual-coding  theory  advanced  by  Paivio 
(1971,  1978). 

According  to  the  dual-coding  theory,  inputs  from  verbal  and  nonverbal  sources  are 
represented  and  processed  by  means  of  two  functionally  independent,  yet  interconnected, 
cognitive  systems.  The  image  system  is  presumed  to  specialize  in  the  processing  of 
pictorial  information  and  in  generating  and  analyzing  mental  images.  The  verbal  system 
is  concerned  with  the  perceptual  processing  and  production  of  language.  The  theory  holds 
that  each  system  can  be  independently  accessed  by  relevant  stimuli:  the  image  system  is 
activated  more  directly  by  pictures,  line  drawings,  or  objects,  while  the  verbal  system  is 
activated  more  directly  by  written  or  spoken  words.  In  addition,  the  theory  assumes  that 
both  systems  are  substantially  interconnected,  since  nonverbal  information  can  be 


9 


transformed  into  verbal  information  (or  vice  versa)  under  appropriate  conditions.  Paivio's 
theory  also  includes  an  explanatory  model,  consisting  of  three  stages  of  processing,  to 
account  for  the  integration  of  information  from  each  system. 

The  central  predictions  made  by  the  dual-coding  theory  are  that:  (1)  information 
represented  by  both  image  and  verbal  codes  is  more  powerful  than  that  represented  by 
either  code  alone,  but  (2)  no  advantage  will  accrue  to  the  presentation  of  redundant 
information  within  the  same  coding  system.  Paivio  and  his  associates  conducted  numerous 
studies  they  thought  supported  these  predictions.  References  to  this  research  are  listed  in 
an  edited  volume  by  Yuille  (1983)  and  in  Paivio  (1978).  Other  researchers,  independent  of 
Paivio  and  his  associates,  have  obtained  results  that  generally  support  the  dual-coding 
concept  (e.g.,  Moyer,  1973;  Rabinowitz,  Mandler,  <5c  Barsalou,  1979;  Rafnel  &  Klatsky, 
1978;  Tversky,  1969). 

Most  of  the  persistent  controversy  about  this  theory  involves  a  debate  between 
imagery  and  propositional  theorists.  As  noted  by  Anderson  (1978),  imagery  theorists 
contend  that  separate  coding  systems  are  used  to  process  verbal  and  visual  information. 
Propositional  theorists,  on  the  other  hand,  contend  that  both  verbal  and  visual  information 
are  coded  using  the  same  abstract,  amodal  format.  The  model  used  by  Paivio  to  explain 
his  dual-coding  theory  was  also  criticized  by  Desrochers  and  Petrusic  (1983)  because  it 
makes  no  provision  for  the  temporary  storage  of  stimuli  in  either  the  verbal  or  image 
systems,  nor  does  it  specify  the  mechanism  for  selecting  responses  once  stimuli  have  been 
processed  through  the  various  stages. 

Summary 

Taken  individually,  the  studies  in  this  review  often  present  conflicting  and  inconsis¬ 
tent  findings  concerning  the  effects  of  audio  and  visual  presentation  methods  on  learning 
and  performance.  Reasonable  consistency  can  be  found,  however,  when  studies  in  each 
content  area  are  considered  together,  and  particularly  when  the  results  from  these  studies 
are  examined  in  the  context  of  the  various  theoretical  formulations  reviewed.  According¬ 
ly,  the  following  paragraphs  summarize  the  major  findings  and  conclusions  derived  from 
the  publications  included  in  each  content  area. 

One  generally  consistent  finding  among  studies  in  academic  settings  was  the 
superiority  of  instructional  material  presented  visually  to  the  same  material  presented 
aurally.  To  account  for  this  finding,  several  authors  (e.g.,  Chapanis,  1965;  Day  &  Beach, 
1950;  Gropper,  1963;  Levie  &  Dickie,  1973)  have  suggested  that  one  advantage  of 
information  presented  visually  is  the  relatively  greater  referability  (i.e.,  opportunity  for 
reviewing  the  material)  that  it  affords.  Thus,  for  example,  literate  adults  using  textual 
materials  can  often  obtain  more  information  per  unit  of  time,  either  by  scanning  ahead 
for  particular  details  or  by  re-reading  the  material  in  whole  or  part,  than  subjects 
receiving  the  same  material  aurally.  Goldstein  (1940)  found  that  the  less  the  referability 
afforded  by  a  visual  presentation,  the  less  its  advantage  over  an  auditory  presentation. 

Other  authors  have  suggested  that  personal  experience  with  information  presented  in 
visual  or  auditory  forms  may  play  a  role  in  determining  the  extent  to  which  one  method  is 
superior  to  the  other.  The  influence  of  this  factor  in  the  context  of  job  performance 
situations  was  discussed  by  Henneman  and  Long  (1954,  p.  16)  who  noted  that  "in  the  past, 
an  operator  who  has  received  simple  direct  instructions  and  orders,  as  spoken  by  others, 
may  well  respond  more  efficiently  to  such  messages  when  presented  by  voice  than  when 
presented  visually,  in  spite  of  many  other  considerations."  Similar  views  were  expressed 
by  Travers  (1970),  but  his  explanation  was  based  on  the  underdeveloped  reading  skills 
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among  younger  subjects  in  the  context  of  academic  settings.  In  summary,  none  of  the 
literature  reviewed  involving  adult  subjects  suggested  the  presence  of  any  special  ability 
to  extract  information  from  either  visual  or  auditory  forms. 

Two  of  the  studies  reviewed  in  applied  settings  showed  that  a  combined  text-pictorial 
format  resulted  in  performance  superior  to  that  resulting  from  the  use  of  text  or  pictorial 
instructions  alone;  while  two  other  studies,  which  used  only  dual-  or  multiple-channel 
presentations,  found  essentially  no  differences  between  the  various  methods  tested.  The 
studies  reviewed  by  Chalupsky  and  Kopf  (1967)  compared  job  performance  when  using  an 
audiovisual  aid  with  previously  established  production  standards  (e.g.,  the  number  of 
acceptable  units  produced  in  a  given  time  period).  While  the  findings  of  these  studies  are 
reasonably  consistent,  they  are  too  few  in  number  to  permit  any  meaningful  generaliza¬ 
tions.  Other  considerations,  such  as  inadequate  specification  of  relevant  treatment 
comparisons  and  a  lack  of  experimental  controls,  also  limit  the  overall  generality  of  the 
findings  and  conclusions  obtained  in  several  cases.  For  example,  in  the  research  reported 
by  Brown  (1964),  Chalupsky  and  Kopf  (1967),  and  Goff  et  al.  (1969),  no  provisions  were 
made  for  separating  the  text  or  audio  instructions  from  the  information  presented 
pictorially.  Thus,  there  is  no  way  of  determining  from  these  studies  whether  the  text  or 
audio  instructions,  either  separately  or  combined,  would  have  provided  sufficient  informa¬ 
tion  with  which  to  perform  the  job  tasks. 

In  a  related  vein,  no  provisions  were  made  in  the  Goff  et  al.  study  for  equating  the 
text-pictorial  and  audio-pictorial  methods  in  terms  of  the  amount  of  information 
presented  at  any  one  time.  Subjects  using  the  text-pictorial  method  could  view  any  or  all 
of  the  performance  steps  listed  in  booklet  form,  while  those  using  the  audio-pictorial 
method  were  presented  with  only  a  single  performance  step  at  a  time.  Further,  subjects 
who  used  written  instructions  could  self-pace  the  reading  of  the  material,  whereas 
subjects  in  the  audio-pictorial  method  were  paced  by  the  rate  at  which  the  narrator  spoke. 
These  problems  notwithstanding,  the  studies  by  Brown  (1964)  and  Goff  et  al.  (1969)  found 
consistent  performance  differences  between  subjects  as  a  function  of  prior  training  and 
experience  on  the  tasks  sampled.  For  that  reason,  it  was  felt  that  this  factor  had  direct 
implications  for  the  design  of  the  present  study. 

One  area  in  which  the  predictions  from  the  various  theories  appear  to  converge 
concerns  the  simultaneous  presentation  of  redundant  information  in  text  and  audio  forms. 
Both  the  limited-capacity  theory  and  the  dual-coding  theory  predict  that  redundant 
information  presented  in  a  text-audio  format  offers  no  advantage  over  the  same  material 
presented  in  either  text  or  audio  form  alone.  It  is  important  to  note,  however,  that  the 
theories  differ  as  to  why  this  is  true.  According  to  limited-capacity  theory,  redundancy 
between  two  separate  information  sources  is  reduced  by  a  process  of  categorization;  that 
is,  some  bits  of  information  are  treated  as  being  equivalent  by  the  subject  and  are 
"chunked  together"  into  fewer  bits  of  information.  The  compressed  information  then 
enters  a  central  processing  system  that  can  handle  information  presented  sequentially  but 
not  simultaneously.  Dual-coding  theory,  in  contrast,  states  that,  while  text  and  audio 
inputs  are  received  via  separate  sensory  modalities,  each  can  be  simultaneously  processed 
by  activating  the  verbal  system,  which  structures  these  types  of  inputs  in  the  same  way- 
through  words.  It  is  only  when  verbal  and  pictorial  representations  of  related  or 
redundant  information  are  used  together  that  people  can  use  each  to  its  best  advantage. 

Another  perspective  on  this  issue  was  offered  in  a  series  of  articles  by  Hower  Hsia 
(1968b,  1969,  1971,  1977),  which  describe  his  interpretation  and  extension  of  Shannon  and 
Weaver's  (1949)  mathematical  theory  of  communication.  According  to  this  theory, 
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communication  systems  usually  employ  a  redundancy  principle  to  reduce  or  minimize  the 
adverse  effects  of  error,  equivocation,  and  noise.  Hsia  (1977)  distinguishes  between  two 
types  of  redundancy:  one  involves  the  content  of  information  within  a  single  channel;  the 
other,  shared  information  between  any  dual-  or  multiple-channel  presentation.  Regarding 
the  latter,  Hsia  provides  an  algebraic  formula  for  computing  the  precise  level  of  between- 
channel  redundancy  (BCR).  The  end  result  of  this  formula  is  expressed  on  a  bivariate 
continuum:  BCR  is  optimum  when  two  or  more  channels  transmit  identical  information 
(i.e.,  producing  mutual  facilitation);  and  it  is  zero  when  the  channels  emit  completely 
different  information  (i.e.,  producing  mutual  interference). 

Hsia  (1977)  proposes  two  factors  that  he  feels  may  account  for  the  many  discrep¬ 
ancies  regarding  single-  versus  dual-  or  multiple-channel  presentations  of  redundant 
information.  First,  he  contends  that  many  experimenters  have  chosen  to  ignore 
quantification  of  the  term  redundancy  via  the  mathematical  formulations  developed  by 
Shannon  and  Weaver;  instead,  they  have  preferred  to  define  it  in  terms  of  common  usage, 
in  which  redundant  relationships  are  interpreted  on  purely  intuitive  grounds.  Second,  he 
notes  that  unless  stimuli  of  the  same  class  and  type  are  synchronized,  any  lagging  or 
staggering  may  interfere  with  the  orderly  processing  of  the  information  transmitted.  If 
this  problem  existed  in  some  of  the  studies  reviewed,  it  could  account  partially  for  the 
often  inconsistent  findings  reported  for  dual-  or  multiple-channel  presentations  of 
instructional  materials  as  compared  with  single-channel  presentations. 

Plan  of  the  Present  Research 


A  diverse  set  of  stimulus  materials,  subjects,  experimental  methods,  and  theoretical 
formulations  have  been  used  in  previous  studies  for  assessing  or  explaining  the  effects  of 
audiovisual  presentations  on  human  learning  and  performance.  The  rationale  used  for 
selecting  the  six  presentation  methods  in  this  study  was  that  they  were  generally 
representative  of  the  types  of  methods  that  either  had  in  the  past,  or  could  in  the  future, 
be  used  for  presenting  the  information  needed  to  perform  job  tasks.  The  experience 
factor  was  selected  because  earlier  research  found  that  it  accounted  for  substantial 
variance  in  task  performance  measures.  Several  experimental  hypotheses  tested  in  this 
study  were  based  on  predictions  suggested  by  Paivio's  (1971,  1978)  dual-coding  theory. 
This  formulation  was  selected  because  it  provided  differential  predictions  regarding  the 
effectiveness  of  the  six  presentation  methods  chosen  for  comparison. 
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METHOD 


Subjects 


Subjects  were  180  Navy  enlisted  personnel  from  the  Naval  Training  Center,  San 
Diego,  California.  The  subjects  ranged  from  17  to  34  years  in  age,  had  between  9  and  16 
years  of  civilian  education,  and  had  between  2  and  36  months  of  military  service.  Two 
groups  of  subjects  were  formed:  one  included  72  males  and  18  females  who  had  completed 
the  oscilloscope  operator  portion  of  a  Basic  Electricity  and  Electronics  (BE/E)  training 
course  and  the  other  included  90  male  recruits  who  had  been  designated  to  receive  BE/E 
training  but  who  had  not  yet  started  the  course.2  For  each  group,  subjects  were  randomly 
ordered  into  one  of  the  six  experimental  treatment  conditions,  thus  there  were  15  subjects 
per  condition. 

Participation  in  the  testing  was  strictly  voluntary.  No  one  declined  to  participate, 
and  all  subjects  appeared  interested  and  cooperative  throughout  the  experimental 
sessions. 

Performance  Tasks 


The  following  four  performance  tasks  involving  the  operation  of  a  standard  Navy 
oscilloscope  were  used  in  this  study: 

Initial  Setup  and  Adjustments 

Subjects  performed  a  33-step  procedure  to  prepare  the  oscilloscope  for  operation. 
The  content  of  instructions  for  this  task  was  adapted  from  a  Navy  oscilloscope  operator 
manual  (Naval  Sea  Systems  Command,  1967).  In  order  to  meet  the  criterion  for  success, 
ninety-five  percent  of  the  steps  had  to  be  performed  completely,  correctly,  and  in  the 
sequence  specified  in  the  instructions. 

Probe  Calibration 


Subjects  performed  a  20-step  procedure  to  calibrate  a  divider  probe.  The  content  of 
instructions  for  this  and  the  two  remaining  tasks  was  adapted  from  an  oscilloscope 
proficiency  test  developed  under  Navy  contract  (Kinton,  Inc.,  1980).  The  criterion  for 
successful  performance  was  defined  as  a  perfect  square  waveform  being  presented  on  the 
oscilloscope  display  screen  subsequent  to  tightening  of  the  probe-locking  collar. 

Amplitude  Measurement 


Subjects  performed  an  18-step  procedure  to  measure  the  peak-to-peak  amplitude  of  a 
test  signal.  They  were  also  required  to  write  down  certain  readings  and  control  settings 
from  the  oscilloscope  and  to  perform  simple  arithmetic  in  computing  an  amplitude  value. 


2  As  initially  planned,  each  subject  group  was  to  consist  of  90  male  Navy  enlisted 
personnel.  This  requirement  was  changed  in  the  case  of  the  BE/E  group,  however,  when  it 
was  learned  that  approximately  20  percent  of  the  students  enrolled  in  that  course  were 
females.  Accordingly,  three  female  BE/E  students  were  assigned  at  random  to  each  of 
the  six  treatment  conditions.  Comparable  numbers  of  female  subjects  were  not  available 
for  testing  in  the  other  group,  since  recruit  training  for  female  Navy  enlisted  personnel  is 
not  provided  in  San  Diego. 
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The  acceptable  range  of  accuracy  was  defined  as  a  measurement  between  11.6  and  12.0 
volts  peak-to-peak. 

Frequency  Measurement 

Subjects  performed  a  17-step  procedure  to  measure  the  frequency  of  a  test  signal. 
This  task  also  required  oscilloscope  readings  and  control  settings  to  be  recorded  and 
arithmetic  computations  to  be  performed.  The  acceptable  range  of  accuracy  was  defined 
as  a  measurement  between  6,944  and  7,353  hertz. 

The  foregoing  tasks  met  several  important  criteria  in  that  they  sampled  a  wide  range 
of  behaviors  having  practical  importance  to  Navy  jobs  concerned  with  the  maintenance  of 
electrical  and/or  electronic  equipment;  required  no  specialized  knowledge,  skills,  or 
reference  materials  for  their  solution;  and  had  definitive,  predetermined  criteria  for 
defining  passing  or  failing  performance. 

Presentation  Methods 


Six  methods  for  presenting  computer-based  instructions  were  tested,  namely: 

1.  Text-only,  in  which  steps  for  each  task  consisted  of  short  statements  presented 
on  a  computer  display  monitor. 

2.  Audio-only,  in  which  the  same  task  steps  were  presented  aurally  from  a 
prerecorded  sound  track. 

3.  Text-audio,  in  which  the  above  methods  were  combined  so  that  the  text  and 
audio  instructions  were  presented  simultaneously. 

4.  Text-graphics,  in  which  the  text  instructions  were  supplemented  with  outline 
drawings  of  the  oscilloscope  and  related  support  equipment.  In  addition,  "flashing  arrows" 
were  used  on  these  drawings  to  highlight  the  particular  switch,  dial,  or  control  described 
in  the  instructions.  Figure  2  gives  an  example  of  a  task  step  presented  by  this  method. 

5.  Audio-graphics,  in  which  task  steps  were  presented  aurally,  supplemented  with 
the  same  outline  drawings  described  above. 

6.  Text-audio-graphics,  in  which  text  and  audio  instructions  were  presented  simul¬ 
taneously,  supplemented  with  outline  drawings. 

Written  instructions  for  the  text-only  and  text-audio  methods  usually  comprised  two 
or  three  lines  of  text  with  a  maximum  of  60  characters  per  line,  while  those  for  the  text- 
graphics  and  text-audio-graphics  methods  were  usually  four  or  five  lines  of  text  with  a 
maximum  of  32  characters  per  line.  As  shown  in  Figure  2,  graphics  appeared  either  below 
or  to  the  left  of  the  text,  which  was  the  configuration  used  in  the  text-graphics  and  text- 
audio-graphics  methods.  This  same  configuration,  less  the  text,  was  used  in  the  audio¬ 
graphics  method.  For  all  but  the  audio-only  method,  the  text  and/or  graphics  information 
appeared  in  phosphorescent  green  against  a  uniform  gray  background  on  the  computer 
display  screen. 
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Step  86. 

Remove  the  probe  ground  lead 
(alligator  clip)  from  TEST 
POINT  3  on  the  black  box 


Figure  2.  An  example  of  a  task  step  presented  by  the  text-graphics  method. 


Instructions  for  presentation  methods  that  included  audio  were  recorded  at  an 
average  rate  of  100  words  per  minute,  which  is  well  below  the  average  oral  reading  rate 
(M  =  175;  SD  =  20)  for  newscasters  and  professional  readers  for  the  blind  (Foulke  <5c  Sticht, 
1969).  The  same  100-words-per-minute  rate  was  adopted  as  the  standard  for  presenting 
instructions  in  all  conditions.  Thus,  the  text  and/or  graphics  information  appeared  on  the 
computer  display  only  for  as  long  as  it  took  to  present  the  entire  performance  step 
aurally.  A  computerized  time-code  system  was  used  to  ensure  that  the  rate  of  presenting 
all  task  steps  was  constant  across  conditions. 

Apparatus 

The  apparatus,  depicted  in  Figure  3,  consisted  of  the  following:  (1)  a  microcomputer 
and  peripherals,  (2)  a  random-access  audio  disk  player/recorder,  (3)  an  oscilloscope  and 
associated  support  equipment,  and  (4)  videotape  equipment.  Each  is  described  in  more 
detail  below. 
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Figure  3.  Arrangement  of  apparatus  in  the  mobile  laboratory. 
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Microcomputer  System 


This  system  was  used  for  presenting  instructions  in  all  six  conditions  and  for 
automatically  recording  certain  dependent  variable  measures.  It  consisted  of  a  micro¬ 
computer  (IBM  model  XT)3  equipped  with  an  internal  clock  card,  an  asynchronous 
communications  card,  and  a  monochromatic  graphics/printer  interface  card  (Hercules 
model  HGC).  The  computer  was  also  connected  to  a  standard  personal  computer  keyboard 
and  monochromatic,  17  x  25  cm  display  monitor  (both  IBM  products). 

Two  keys  on  the  computer  keyboard  were  programmed  as  "command"  keys.  These 
keys,  which  occupied  the  numeric  characters  one  and  zero  on  the  uppermost  keyboard 
row,  were  used  by  subjects  for  advancing  to  the  next  instruction  in  the  task  sequence  or 
for  repeating  the  most  recently  presented  one.  The  numeral  one  and  zero  keys  had 
affixed  "ADVANCE"  and  "REPEAT"  labels,  respectively,  to  facilitate  their  identification 
by  the  subjects.  The  computer  was  also  programmed  to  record  automatically  each 
subject's  start  and  stop  times  and  the  number  of  steps  repeated  for  each  task.  Listings  of 
these  data  were  obtained  from  a  line  printer  (Qume  model  Sprint-11)  connected  to  the 
microcomputer. 

Audio  Disk  System 

This  system  was  used  to  present  instructions  for  the  four  conditions  that  used  audio. 
It  consisted  of  a  random-access  audio  disk  player/recorder  (INSTAVOX  model  RA-12A) 
that  was  connected  to  the  microcomputer  and  to  an  external  speaker  (Grason-Stadler 
model  E700A-2).  The  instructions  recorded  on  this  system  were  presented  at  an  average 
volume  level  of  75  decibels. 

Oscilloscope  and  Related  Equipment 

This  system  was  used  to  support  the  subject's  performance  of  the  tasks.  It  consisted 
of  an  oscilloscope  (Tektronix  model  AN/USM-281),  a  10-megaohm  divider  probe  (Tektronix 
model  P6006),  a  test  signal  generator  (Continental  Specialties  model  2001),  a  six-function 
pocket  calculator  (Light  Power),  and  a  "black  box"  apparatus  that  was  designed  for  an 
earlier  research  project  involving  the  AN/USM-281  oscilloscope  (Nugent,  Laabs,  &  Panell, 
1982). 

External  labels  were  affixed  or  modified  on  three  of  the  above  equipments  to  identify 
various  components  that  subjects  were  required  to  use  while  performing  the  tasks.  For 
example,  the  following  labels  were  affixed  to  components  of  the  divider  probe:  PROBE 
BODY  AND  TIP,  LOCKING  COLLAR,  and  BNC  CONNECTOR.  These  labels  corresponded 
exactly  with  those  used  to  name  the  components  in  the  task  steps.  A  similar  procedure 
was  used  to  label  components  on  the  black  box  and  oscilloscope. 

Videotape  System 

This  system  was  used  to  videotape  performances  of  a  sample  of  subjects  in  each  of 
the  12  combinations  of  subject  group  and  treatment  conditions.  It  consisted  of  a  low 
light-level  television  camera  and  video  processor  unit  (Sony  model  AVC-1450),  a  videotape 
recorder  unit  (Sony  model  SLO-320),  and  a  television  monitor  (Sony  model  CVM-115). 


3  Identification  of  the  equipment  used  in  this  study  is  for  documentation  purposes  only 
and  does  not  imply  endorsement. 
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Procedure 


The  subjects  were  tested  individually  in  a  large  mobile  laboratory  belonging  to  the 
Navy  Personnel  Research  and  Development  Center,  San  Diego,  California.  Instructions 
were  read  aloud  to  the  subject  at  the  outset  of  the  session  to  provide  a  general  orientation 
to  the  performance  tasks  and  data  collection  methods.  Voluntary  participation  was 
explained  and  procedures  for  maintaining  subject  anonymity  were  described,  followed  by 
distribution  of  a  background  information  form  (Appendix  A). 

After  a  subject  was  randomly  assigned  to  a  presentation  method,  standardized 
instructions  specific  to  that  method  were  read  aloud.  The  subject  next  performed  a  four- 
step  practice  exercise  that  was  designed  to  provide  familiarization  with  the  method  of 
presenting  task  instructions,  the  types  of  actions  to  be  performed,  and  the  use  of  the 
command  keys. 

Instructions  for  each  oscilloscope  operator  task  were  then  read  aloud,  with  subjects 
performing  them  in  the  following  fixed  order:  initial  setup  and  adjustments,  probe 
calibration,  amplitude  measurement,  and  frequency  measurement.1*  Each  set  of  instruc¬ 
tions  began  with  a  statement  describing  the  purpose  of  the  task  to  be  performed,  followed 
by  supplementary  information  concerning  such  items  as  appropriate  use  of  the  command 
keys  and  where  to  record  various  readings  and  values  obtained.  These  instructions  also 
advised  the  subject  that  the  experimenter  would  not  answer  questions  or  provide 
assistance  during  performance  of  the  task,  but  that  the  subject  would  be  informed  when 
the  task  had  been  completed.  Equipment  settings  and  adjustments  were  standardized 
prior  to  the  administration  of  each  task.  An  example  of  relevant  equipment  control 
settings  used  prior  to  each  subject's  performance  of  the  probe  calibration  task  is  given  in 
Appendix  B. 

Data  Collection  Forms 


Three  basic  forms  were  used  for  recording  subjects'  task  performance  and  attitudinal 
data.  One  of  these,  termed  the  performance  observation  form,  was  used  by  the 
experimenter  to  record  errors  that  subjects  made  during  their  performance  of  the  tasks. 
A  zero  value  was  assigned  for  task  steps  that  were  performed  incorrectly  or  out  of 
sequence  (omitted  steps  were  considered  as  being  performed  both  incorrectly  and  out  of 
sequence).  Spaces  were  also  provided  on  the  forms  for  recording  the  specific  reason(s) 
why  a  zero  value  was  assigned  to  any  given  task  step.  Appendix  C  provides  an  example  of 
the  observation  form  used  to  assess  each  subject's  performance  during  the  initial  setup 
and  adjustment  task. 

The  second  form,  termed  the  examinee  worksheet,  was  completed  by  the  subject 
during  his  or  her  performance  of  the  amplitude  and  frequency  measurement  tasks.  This 
form  contained  spaces  for  recording  oscilloscope  readings  and  the  results  of  various 
arithmetic  computations  (see  Appendix  D). 


Counterbalancing  for  task  order  effects  was  rejected  because  such  an  approach 
could  result  in  potential  damage  to  the  oscilloscope  and  its  associated  support  equipment. 
Moreover,  the  particular  task  presentation  order  used  is  the  only  one  that  is  consistent 
with  standard  Navy  procedures  for  the  operation  of  oscilloscopes. 
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The  third  form,  a  user  evaluation  questionnaire,  was  completed  by  each  subject 
immediately  after  the  frequency  measurement  task  had  been  completed.  The  first  section 
of  the  questionnaire  contained  items  for  rating  various  physical  features  of  the  equipment 
used  to  support  the  subject's  performance  of  the  tasks,  while  the  second  section  involved 
rating  the  acceptability  of  the  method  used  to  present  the  task  instructions.  The  third 
section  provided  spaces  for  making  comments,  complaints,  or  suggestions  (Appendix  E). 

Video  Recordings 

Task  performances  were  recorded  on  videotape  for  36  (20%)  of  the  180  subjects 
tested.  This  subset  was  formed  by  selecting  3  subjects  from  each  of  the  12  combinations 
of  subject  groups  and  treatment  conditions.  Within  each  of  these  combinations,  1  subject 
was  selected  at  random  from  each  one-third  of  the  15  subjects  tested  (i.e.,  1  subject  from 
the  first-,  middle-,  and  last-third,  respectively).  Videotaping  established  a  permanent 
record  of  oscilloscope  operator  task  performances  for  subsequent  review  and  analysis. 

Debriefing 


The  test  session,  which  lasted  an  average  of  59  minutes  per  subject,  ended  with  a 
discussion  of  the  purpose  of  the  study,  together  with  a  thorough  review  of  the  subject's 
performance  on  the  tasks.  Subjects  were  thanked  for  participating  and  asked  to  refrain 
from  discussing  details  of  the  study  with  others  who  had  yet  to  be  tested. 

Experimental  Hypotheses  and  Design 

The  following  hypotheses  served  as  the  framework  for  comparing  task  performance  as 
a  function  of  subject  group  and  presentation  method: 

I.  As  a  group,  BE/E  subjects  will  demonstrate  better  task  performance  (defined  as 
faster  time  to  completion,  more  tasks  performed  correctly,  fewer  steps  repeated,  fewer 
steps  performed  incorrectly,  and  fewer  steps  performed  out  of  sequence)  than  the  recruit 
subjects. 


II.  For  both  groups,  subjects  assigned  to  presentation  methods  that  include  graphics 
will  demonstrate  better  task  performance  than  subjects  assigned  to  methods  that  exclude 
graphics. 

III.  For  both  groups,  no  difference  in  task  performance  will  be  found  between 
subjects  assigned  to  the  text-audio  method  and  subjects  assigned  to  the  text-only  and 
audio-only  methods. 

IV.  For  both  groups,  no  difference  in  task  performance  will  be  found  between 
subjects  assigned  to  the  text-audio-graphics  method  and  subjects  assigned  to  the  text- 
graphics  and  audio-graphics  methods. 

The  independent  variables  of  subject  group  and  presentation  method  were  arranged  in 
a  2  X  6  between-subjects  multiple  analysis  of  variance  (MANOVA)  design  for  preliminary 
analyses  of  the  data.  In  subsequent  analyses,  this  design  was  reduced  to  a  series  of  single¬ 
factor,  between-subjects  MANOVAs  to  test  the  validity  of  the  experimental  hypotheses 
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described  above.  Additionally,  five  categories  of  performance  measures  were  included  in 
the  analyses  of  the  oscilloscope  operator  tasks  taken  both  separately  and  overall  (i.e., 
summed  across  tasks).  The  specific  variables  that  comprised  these  five  categories 
included:  time-to-task  completion,  dichotomous  (pass  or  fail)  performance  scores,  and 
separate  frequency  counts  for  task  steps  that  were  repeated,  performed  incorrectly,  or 
performed  out  of  sequence.  Taken  together,  the  four  sets  of  five  outcome  measures  for 
the  separate  tasks  and  the  one  set  of  five  overall  task  performance  measures  yielded  a 
total  of  25  dependent  variables  for  use  in  subsequent  data  analyses. 


RESULTS 

The  means  and  standard  deviations  for  the  five  categories  of  performance  measures 
used  in  this  study  (cross-tabulated  by  tasks,  groups,  and  presentation  methods)  are 
presented  in  Table  1.  The  individual  scores  upon  which  Table  1  is  based  were  submitted  to 
the  MANOVA  designs  described  in  the  previous  section. 

Preliminary  Analyses 

Four  separate  analyses  were  conducted  preparatory  to  the  main  tests  of  the 
experimental  hypotheses.  The  purposes  of  these  analyses  were  to  determine:  (1)  whether 
a  statistically  significant  interaction  existed  between  the  subject  grouping  and  presenta¬ 
tion  method  factors,  (2)  whether  performance  differences  existed  between  the  male  and 
female  subjects  who  comprised  the  BE/E  group,  (3)  whether  subjects  included  in  each 
group  and  presentation  method  were  comparable  in  terms  of  certain  background  charac¬ 
teristics,  and  (4)  whether  these  background  characteristics  warranted  inclusion  as 
covariates  in  subsequent  data  analyses. 

Interaction  Effects 


Because  all  main  tests  of  the  experimental  hypotheses  were  necessarily  limited  to  a 
single  between-subjects  factor,  the  first  analysis  was  particularly  important  in  determin¬ 
ing  whether  the  effect  of  the  subject  grouping  factor  was  the  same  across  all  levels  of  the 
presentation  method  factor.  Accordingly,  performance  outcome  measures  for  all  subjects 
were  included  in  the  2X6  between-subjects  MANOVA  design  described  earlier.  Results 
showed  a  nonsignificant  multivariate  F-ratio  for  the  interaction  between  these  two 
factors  on  the  separate  (F(100,  765)  =  1.12,  jj  <  .20)  and  overall  (F(25,  840)  =  1.00,  p<  .50) 
task  measures.  This  analysis  did,  however,  yield  significant  multivariate  F-ratios  for  the 
main  effect  of  each  factor  separately,  results  of  which  are  discussed  later  in  this  section. 


A  split-plot  MANOVA  design  that  would  include  the  four  tasks  as  a  within-subjects 
(repeated  measures)  factor  was  considered  but  rejected  as  inappropriate.  The  principal 
reason  for  rejecting  this  design  concerned  the  lack  of  parallelism  among  tasks  in  terms  of 
their  difficulty  level  and  the  number  and  type  of  actions  each  required. 
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(2.0)  (3.6)  (0.9)  (1.2)  (0.4) 


8.3  2.3  2.2  0.7  0.5 
(2.0)  (2.5)  (1.9)  (0.8)  (0.5) 


8.4  1.9  2.1  0.2  0.6 

(2.0)  (2.0)  (2.2)  (0.6)  (0.5) 


33.8  11.1 

(6.0)  (10.1) 


6.2  2.7  2.7 

(3.7)  (2.3)  (1.0) 


aStandard  deviations  enclosed  in  parentheses. 
^Legend:  T  -  Text,  A  =  Audio,  G  -  Graphics. 


Gender  Effects 


The  second  analysis  compared  male  and  female  BE/E  students  in  terms  of  their 
performance  on  the  separate  and  overall  tasks.  Specifically,  in  the  event  that  the  task 
performance  of  the  18  female  BE/E  students  was  found  to  be  superior  to  that  of  the  72 
male  BE/E  students  (or  vice  versa),  such  an  effect  would  have  to  be  accounted  for  either 
by  including  the  subject's  gender  as  an  additional  factor  in  subsequent  analyses  of  the 
BE/E  group  data,  or  by  removing  the  18  female  subjects  altogether,  thereby  requiring  a 
commensurate  reduction  in  the  recruit  sample  to  maintain  equal  cell  sizes  in  each 
presentation  method.  These  procedures  were  not  needed,  however,  since  the  multivariate 
F-ratios  for  the  interaction  between  the  gender  and  presentation  method  factors  and  the 
main  effect  for  the  gender  variable  failed  to  reach  statistical  significance  on  both  the 
separate  and  overall  task  measures. 

Comparability  on  Background  Variables 

The  third  analysis  focused  on  the  comparability  of  the  12  groups  of  subjects  in  terms 
of  the  following  personal  background  variables:  age,  years  of  formal  education,  and  prior 
civilian  schooling  or  work  experience  in  electricity  or  electronics.6  To  accomplish  this, 
three  separate  univariate  analyses  of  variance  (ANOVAs)  were  performed  in  which  the 
above  background  characteristics  served  as  dependent  variables,  with  the  subject  group 
and  presentation  methods  serving  as  independent  variables.  Results  showed  nonsignificant 
F-ratios  for  the  main  effects  of  the  two  between-subjects  factors  and  their  associated 
two-way  interactions  in  all  three  ANOVAs. 

Potential  Covariates 


This  analysis  compared  performance  on  separate  and  overall  tasks  using  the  above 
background  characteristics  as  2-level  factors  in  separate  2X2X6  between-subjects 
M ANOVAs.  The  two  factor  levels  for  the  age  and  education  variables  were  formed  by 
dividing  each  distribution  at  its  respective  median,  while  the  third  background  variable 
(i.e.,  prior  civilian  schooling  or  work  experience  in  electricity/electronics)  assumed  the 
same  dichotomous  values  described  in  footnote  6.  MANOVA  results  showed  nonsignificant 
F-ratios  for  all  interaction  terms  and  main  effects  for  these  background  variables  on  the 
separate  and  overall  task  measures;  consequently  their  use  in  subsequent  analyses  was 
abandoned. 

Main  Analyses 

The  following  four  sections  present  results  from  the  main  tests  of  the  experimental 
hypotheses.  The  reader  should  refer  to  Table  1  for  a  listing  of  the  mean  values  compared 
in  each  of  these  tests. 


6The  latter  variable,  which  was  based  on  the  subject's  responses  to  questions  8  and  9 
on  the  Background  Information  Form,  was  coded  "0"  if  both  questions  were  answered  in 
the  negative  and  "1"  if  either  or  both  questions  were  answered  in  the  affirmative.  Other 
items  on  the  Background  Information  Form  were  excluded  from  further  consideration 
because  of  insufficient  variability  in  the  distribution  of  subjects'  responses. 
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Hypothesis  I 


This  hypothesis  predicted  better  task  performance  by  the  BE/E  subjects  than  by  the 
recruit  subjects.  Relevant  data  were  analyzed  in  five  MANOVAs  (one  each  for  the 
separate  and  overall  task  measures),  which  included  the  subject  groups  as  two  levels  of 
the  between-subjects  factor.  A  summary  of  the  results  obtained  in  these  analyses  is 
presented  in  Table  2.  Part  I  of  this  table  shows  significant  multivariate  F-ratios  for  both 
the  separate  and  overall  tasks;  thus,  these  data  provide  a  basis  for  rejecting  the  null 
hypothesis  that  BE/E  and  recruit  subjects  would  perform  equally  well  on  the  tasks.  Given 
these  findings,  it  is  appropriate  to  examine  univariate  F-ratios  to  identify  the  specific 
variables  on  which  the  two  subject  groups  differed. 

Accordingly,  as  shown  in  Table  2,  Part  II,  significant  differences  were  found  between 
the  two  subject  groups  with  respect  to:  (1)  the  time  variable  on  the  setup  task;  (2)  all 
variables  except  sequence  errors  on  the  probe  task;  (3)  all  variables  except  the  success 
criterion  on  the  amplitude  task;  (4)  the  success  criterion  on  the  frequency  task;  and  (5)  all 
variables  except  sequence  errors  on  the  overall  tasks.  With  the  exception  of  sequence 
errors  on  the  probe  task,  the  performance  of  the  BE/E  group  was  superior  to  the  recruit 
group. 

A  relative  index  of  the  strength  of  the  subject-grouping  factor  on  the  various 
performance  outcome  measures  is  provided  in  the  last  column  of  Table  2,  Part  II. 
Inspection  of  the  w2  values  shows  a  range  from  2  to  14  percent.  One  method  for 
interpreting  these  values  was  provided  by  Cohen  (1977,  pp.  284-288),  who  suggested  that 
"a  'large'  effect  in  the  behavioral  and  social  sciences  is  an  experiment  that  produces  a 
w2  value  of  .15;  a  'medium'  effect  is  .06;  and  a  'small'  effect  is  .01."  While  such  arbitrary 
distinctions  can  be  questioned,  this  rough  scale  provides  some  perspective  with  which  to 
interpret  the  strength  of  the  subject-grouping  factor.  Based  on  Cohen's  scale,  prior 
training  and  experience  in  the  operation  of  oscilloscopes  had  only  a  small-to-medium 
effect  on  the  variance  in  the  performance  outcome  measures.  In  summary,  the  foregoing 
results  generally  support  Hypothesis  I  in  terms  of  the  probe,  amplitude,  and  overall  tasks 
but  provide  only  limited  support  for  the  hypothesis  in  the  case  of  the  setup  and  frequency 
tasks. 

Hypothesis  II 

This  hypothesis  predicted  better  task  performance  by  subjects  whose  presentation 
method  included  graphics  than  by  subjects  whose  methods  excluded  graphics.  Relevant 
data  were  analyzed  using  five  MANOVAs  in  which  the  inclusion  or  exclusion  of  graphics 
served  as  the  two  levels  of  the  between-subjects  factor.  Results  from  these  analyses  are 
summarized  in  Table  3. 

Part  I  of  this  table  shows  significant  multivariate  F-ratios  between  presentation 
methods  that  included  graphics  and  those  that  did  not  for  all  but  the  frequency  task.  To 
get  some  idea  of  the  specific  variables  on  which  these  methods  differed,  Table  3,  Part  II, 
shows  significant  univariate  F-ratios  for  all  five  categories  of  measures  used  to  assess 
performance  on  the  setup  task;  the  time  and  repeat  measures  on  the  probe  and  amplitude 
tasks;  and  the  time,  repeat,  and  sequence  measures  on  the  overall  tasks.  The  performance 
of  subjects  using  methods  that  included  graphics  was  superior  to  that  of  subjects  using 
methods  that  excluded  them,  thus  these  data  partially  confirm  the  pattern  of  results 
predicted  by  Hypothesis  II. 
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Table  2 

Recruit  Versus  BE/E  Group  Performance  on  Separate  and  Overall  Tasks 


Part  I :  Multivariate  tests  with  5, 174  degrees  of  freedom  ( df ) . 


Unit  of  Analysis 

Trace  V  a 

F 

E  < 

Setup  Task 

0.081 

3.09 

.05 

Probe  Task 

0.237 

10.81 

.01 

Amplitude  Task 

0.222 

9.91 

.01 

Frequency  Task 

0.154 

6.33 

.01 

Overall  Tasks 

0.673 

6.82 

.01 

Part  II  :  Univariate  tests  with  1,  178  df . 


Setup  Task 


Variable 

Numerator  MS 

Error  MS 

F 

E  < 

Time 

0.637 

0.051 

12.49 

.01 

.06 

Repeats 

2.264 

0.914 

2.47 

.12 

. _ 

Control  Setting  Errors 

0.588 

0.226 

2.60 

.11 

_ 

Sequence  Errors 

0.433 

0.233 

1.86 

.18 

_ _ 

Success  Criterion 

0.672 

0.219 

3.06 

.08 

.... 

Probe  Task 


Variable 

Numerator  MS 

Error  MS 

F 

E  < 

CO2 

Time 

1.590 

0.054 

29.44 

.01 

.14 

Repeats 

5.244 

0.729 

7.19 

.01 

.03 

Control  Setting  Errors 

4.862 

0.201 

24.19 

.01 

.11 

Sequence  Errors 

0.001 

0.182 

0.01 

.94 

Success  Criterion 

0.800 

0.176 

4.54 

.05 

— 

Multivariate  F-ratios  were  based  on  the  trace  V  statistic  described  in  K.C.S.  Pillai 
(1955). 

)  .  9 

For  significant  univariate  F-ratios,  omega  squared  (to  )  provides  a  relative  index  of 

treatment  magnitude  (i.e.,  the  proportion  of  variance  in  outcome  measures  accounted 
for  by  the  independent  variable). 
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Table  2  (Continued) 


Part  II :  Univariate  tests  with  1 , 178  df . 


Amplitude  Task 


Variable 

Numerator  MS 

Error  MS 

F 

E  < 

CO2 

Time 

1.264 

0.048 

26.33 

.01 

.12 

Repeats 

15.677 

0.547 

28.66 

.01 

.13 

Control  Setting  Errors 

5.782 

0.364 

15.88 

.01 

.08 

Sequence  Errors 

0.865 

0.168 

5.14 

.05 

.02 

Success  Criterion 

0.939 

0.247 

3.80 

.06 

— 

Frequency  Task 

Variable 

Numerator  MS 

Error  MS 

F 

E  < 

CO2 

Time 

0.003 

0.050 

0.06 

.82 

_ 

Repeats 

0.442 

0.383 

1.15 

.29 

— 

Control  Setting  Errors 

0.005 

0.382 

0.01 

.91 

— 

Sequence  Errors 

0.076 

0.082 

0.93 

.34 

Success  Criterion 

2.939 

0.236 

12.45 

.01 

.06 

Overall  Tasks 

Variable 

Numerator  MS 

Error  MS 

F 

E< 

CO2 

Time 

0.673 

0.033 

20.39 

.01 

.12 

Repeats 

20.590 

2.142 

9.61 

.01 

.08 

Control  Setting  Errors 

8.721 

0.542 

16.09 

.01 

.08 

Sequence  Errors 

1.384 

0.441 

3.13 

.08 

— 

Success  Criteria 

1.805 

0.107 

16.87 

.01 

.09 
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Table  3 

Comparison  of  Methods  that  Included  Graphics  Versus  Methods 
that  Excluded  Them  on  Separate  Overall  Tasks 


Part  I :  Multivariate  tests  with  5, 1 74  degrees  of  freedom  ( df ). 


Unit  of  Analysis 

Trace  V 

F 

E  < 

Setup  Task 

0.372 

20.59 

.01 

Probe  Task 

0.140 

5.67 

.01 

Amplitude  Task 

0.097 

3.76 

.01 

Frequency  Task 

0.051 

1.87 

.11 

Overall  Tasks 

0.227 

10.24 

.01 

Part  II :  Univariate  tests  with  1,  178  df  ,a 


Setup  Task 


Variable 

Numerator  MS 

Error  MS 

F 

E  < 

<o2 

Time 

3.148 

0.037 

85.08 

.01 

.32 

Repeats 

35.721 

0.726 

49.20 

.01 

.21 

Control  Setting  Errors 

1.692 

0.220 

7.69 

.01 

.04 

Sequence  Errors 

4.362 

0.211 

20.67 

.01 

.10 

Success  Criterion 

1.250 

0.216 

5.79 

.05 

.03 

Probe  Task 

Variable 

Numerator  MS 

Error  MS 

F 

E< 

CO2 

Time 

1.062 

0.057 

18.63 

.01 

.09 

Repeats 

14.720 

0.675 

21.80 

.01 

.10 

Control  Setting  Errors 

0.009 

0.228 

0.04 

.84 

Sequence  Errors 

0.033 

0.181 

0.18 

.67 

Success  Criterion 

0.022 

0.181 

0.12 

.73 

— 

Univariate  results  are  not  listed  for  the  frequency  task  due  to  the  nonsignificant 
F-ratio  obtained  for  it  at  the  multivariate  level  of  analysis  (see  Part  I  above) . 
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Table  3  (Continued) 


Part  II :  Univariate  tests  with  1 , 178  df . 


Amplitude  Task 


Variable 

Numerator  MS 

Error  MS 

F 

E  < 

CO2 

Time 

0.392 

0.053 

7.39 

.01 

.03 

Repeats 

7.133 

0.595 

11.98 

.01 

.06 

Control  Setting  Errors 

1.131 

0.391 

2.89 

.10 

Sequence  Errors 

0.001 

0.172 

0.01 

.97 

— 

Success  Criterion 

0.272 

0.251 

1.08 

.30 

— 

Overall  Tasks 

Variable 

Numerator  MS 

Error  MS 

F 

E< 

CO2 

Time 

1.029 

0.031 

33.19 

.01 

.15 

Repeats 

63.944 

1.898 

33.69 

.01 

.15 

Control  Setting  Errors 

0.075 

0.591 

0.13 

.72 

— 

Sequence  Errors 

3.545 

0.428 

8.28 

.01 

.04 

Success  Criteria 

0.049 

0.117 

0.42 

.52 

— 
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The  magnitudes  of  the  w2  values  listed  in  Part  II  of  Table  3  suggest  that  the  between- 
subjects  factor  included  in  these  analyses  had  a  pronounced  effect  on  the  time  and  repeat 
measures  for  the  setup  and  overall  tasks.  This  effect  diminished,  however,  for  the 
remaining  performance  outcome  measures. 

Hypothesis  III 

This  hypothesis  predicted  no  differences  in  task  performance  among  subjects  whose 
presentation  method  excluded  graphics.  Accordingly,  relevant  data  were  analyzed  in  five 
MANOVAs  in  which  the  presentation  methods  that  excluded  graphics  served  as  three 
levels  of  the  between-subjects  factor.  Results  of  these  MANOVAs  are  summarized  in 
Table  4.  Part  I  of  this  table  shows  statistically  significant  F-ratios  for  both  the  separate 
and  overall  oscilloscope  operator  tasks,  thus  these  data  provide  a  basis  for  rejecting  the 
third  experimental  hypothesis.  Part  II  of  this  table  shows  significant  univariate  F-ratios 
for  the  repeat  measures  in  all  cases  and  for  the  criterion  measure  on  the  setup  task? 

Results  from  application  of  the  Tukey  (1949)  procedure  for  comparing  group  means 
subsequent  to  ANOVA  showed  that  subjects  using  the  text-only  method  repeated  signifi¬ 
cantly  more  task  steps  than  subjects  using  the  audio-only  and  text-audio  methods  (£  <  .0.5 
in  all  cases).  Further,  a  significantly  larger  proportion  of  subjects  using  the  text-only 
method  failed  to  reach  the  criterion  for  successful  completion  of  the  setup  task  than  did 
those  using  the  other  two  methods  (p  <  .05).  In  addition,  the  magnitudes  of  the  w2  values 
listed  in  Part  II  of  Table  4  suggest  that  the  inclusion  or  exclusion  of  auditory  instructions 
had  a  relatively  large  effect  on  the  repeat  measures  for  both  the  separate  and  overall 
tasks,  but  a  somewhat  weaker  effect  for  the  success  criterion  measure  on  the  setup  task. 

Hypothesis  IV 

This  hypothesis  predicted  no  difference  in  task  performance  among  subjects  whose 
presentation  methods  included  graphics.  Relevant  data  were  analyzed  in  five  MANOVAs 
wherein  the  between-subjects  factor  was  represented  by  the  three  presentation  methods 
that  included  graphics.  Results  of  these  analyses  are  presented  in  Table  5.  Part  I  of  this 
table  shows  significant  F-ratios  across  the  separate  and  overall  tasks;  consequently,  these 
data  provide  a  basis  for  rejecting  the  fourth  experimental  hypothesis.  As  shown  in  Part  II 
of  this  table,  significant  univariate  F-ratios  were  obtained  for  the  repeat  measures  in  all 
cases;  for  control  setting  errors  on  the  amplitude  task;  and  for  the  success  criterion  on 
the  probe  and  overall  tasks. 

Application  of  the  Tukey  post-hoc  analysis  procedure  (£  <  .05  in  all  cases)  showed 
that,  on  the  average:  (1)  subjects  using  the  text-graphics  method  repeated  more  steps  on 
the  separate  and  overall  tasks  than  subjects  using  the  other  two  methods;  (2)  more  control 
setting  errors  were  made  on  the  amplitude  task  by  subjects  who  used  the  text-graphics 
method  than  by  those  who  used  the  text-audio-graphics  method;  (3)  a  larger  proportion  of 
subjects  using  the  text-graphics  method  did  not  reach  the  criterion  for  successful 
performance  on  the  probe  task  than  subjects  using  the  audio-graphics  method;  and  (4) 
overall  success  in  performing  the  tasks  was  significantly  higher  for  subjects  using  the 
audio-graphics  and  text-audio-graphics  methods  than  for  subjects  who  used  the  text- 
graphics  method. 

The  magnitudes  of  the  w2  values  listed  in  this  table  are  generally  consistent  with 
those  for  the  previous  MANOVA.  These  results  again  suggest  that  the  inclusion  or 
exclusion  of  auditory  instructions  had  a  pronounced  effect  on  the  repeat  measures 
obtained  for  the  separate  and  overall  tasks,  but  had  only  a  small-to-moderate  effect  on 
the  remaining  performance  outcome  measures. 
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Table  4 

Comparison  of  Methods  that  Excluded  Graphics  on  Separate  and  Overall  Tasks 


Part  I :  Multivariate  tests  with  10, 168  degrees  of  freedom  ( df ). 


Unit  of  Analysis 

Trace  V 

F 

2< 

Setup  Task 

0.495 

5.53 

.01 

Probe  Task 

0.382 

3.97 

.01 

Amplitude  Task 

0.302 

2.99 

.01 

Frequency  Task 

0.212 

1.98 

.05 

Overall  Tasks 

0.459 

5.50 

.01 

Part  II :  Univariate  tests  with  2,  87  df . 


Setup  Task 


V ariable  Numerator  MS  Error  MS 

F 

2  < 

CO2 

Time 

0.020 

0.048 

0.42 

.66 

_ 

Repeats 

11.154 

0.522 

21.36 

.01 

.31 

Control  Setting  Errors 

0.339 

0.193 

1.75 

.18 

— 

Sequence  Errors 

0.373 

0.235 

1.58 

.22 

— 

Success  Criterion 

1.144 

0.224 

5.11 

.01 

.08 

Probe  Task 

Variable  Numerator  MS  Error  MS 

F 

2  < 

CO2 

Time 

0.039 

0.062 

.63 

.53 

_ 

Repeats 

6.406 

0.574 

11.16 

.01 

.18 

Control  Setting  Errors 

0.385 

0.246 

1.56 

.22 

— 

Sequence  Errors 

0.341 

0.178 

1.92 

.16 

— 

Success  Criterion 

0.144 

0.188 

0.76 

.47 

— 

Amplitude  Task 

Variable 

Numerator  MS 

Error  MS 

F 

2  < 

Time 

0.034 

0.056 

0.61 

.54 

_ 

Repeats 

3.943 

0.565 

6.97 

.01 

.12 

Control  Setting  Errors 

0.359 

0.403 

0.89 

.42 

— 

Sequence  Errors 

0.057 

0.155 

0.37 

.70 

_ 

Success  Criterion 

0.211 

0.253 

0.83 

.44 
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Table  4  (Continued) 


Part  II :  Univariate  tests  with  2,  87  df . 


Frequency  Task 


Variable 

Numerator  MS 

Error  MS 

F 

E  < 

CO2 

Time 

0.046 

0.050 

0.92 

.41 

Repeats 

2.698 

0.379 

7.12 

.01 

.12 

Control  Setting  Errors 

1.025 

0.371 

2.76 

.07 

_ 

Sequence  Errors 

0.043 

0.088 

0.49 

.62 

— 

Success  Criterion 

0.411 

0.247 

1.66 

.20 

_ 

Overall  Tasks 

Variable 

Numerator  MS 

Error  MS 

F 

E< 

CO2 

Time 

0.020 

0.036 

0.56 

.57 

_ 

Repeats 

25.243 

1.351 

18.68 

.01 

.28 

Control  Setting  Errors 

0.828 

0.551 

1.50 

.23 

_ 

Sequence  Errors 

0.515 

0.348 

1.48 

.23 

— 

Success  Criteria 

0.217 

0.097 

2.23 

.12 

— 
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Table  5 

Comparison  of  Methods  that  Included  Graphics  on  Separate  and  Overall  Tasks 


Part  I :  Multivariate  tests  with  10,  168  degrees  of  freedom  ( df ). 


Unit  of  Analysis 

Trace  V 

F 

E  < 

Setup  Task 

0.420 

4.47 

.01 

Probe  Task 

0.449 

4.86 

.01 

Amplitude  Task 

0.328 

3.29 

.01 

Frequency  Task 

0.323 

3.24 

.01 

Overall  Tasks 

0.482 

5.33 

.01 

Part  II :  Univariate  tests  with  2,  87  df . 


Setup  Task 


Variable  Numerator  MS  Error  MS 

F 

E  < 

CO2 

Time 

0.002 

0.027 

0.07 

.91 

.... 

Repeats 

7.210 

0.540 

13.35 

.01 

.22 

Control  Setting  Errors 

0.220 

0.245 

0.90 

.41 

_ 

Sequence  Errors 

0.449 

0.177 

2.53 

.09 

_ 

Success  Criterion 

0.278 

0.185 

1.50 

.23 

— 

Probe  Task 

V ariable  Numerator  MS  Error  MS 

F 

E  < 

CO2 

Time 

0.001 

0.054 

0.02 

.98 

Repeats 

6.501 

0.511 

12.72 

.01 

.21 

Control  Setting  Errors 

0.465 

0.201 

2.31 

.11 

Sequence  Errors 

0.064 

0.184 

0.35 

.71 

_ 

Success  Criterion 

0.744 

0.162 

4.59 

.05 

.07 

Amplitude  Task 


Variable 

Numerator  MS 

Error  MS 

F 

E  < 

co2 

Time 

0.064 

0.049 

1.30 

.28 

Repeats 

4.747 

0.452 

10.50 

.01 

.17 

Control  Setting  Errors 

1.480 

0.354 

4.18 

.05 

.04 

Sequence  Errors 

0.463 

0.185 

2.52 

.09 

Success  Criterion 

0.711 

0.239 

2.97 

.06 

— 
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Table  5  (Continued) 


Part  II :  Univariate  tests  with  2,  87  df . 


Frequency  Task 


Variable 

Numerator  MS 

Error  MS 

F 

E  < 

co2 

Time 

0.049 

0.046 

1.06 

.35 

Repeats 

3.208 

0.240 

13.36 

.01 

.22 

Control  Setting  Errors 

0.099 

0.382 

0.26 

.78 

— 

Sequence  Errors 

0.097 

0.077 

1.25 

.29 

— 

Success  Criterion 

0.211 

0.253 

0.83 

.44 

— 

Overall  Tasks 


Variable 

Numerator  MS 

Error  MS 

F 

E< 

CO2 

Time 

0.015 

0.026 

0.57 

.57 

_ 

Repeats 

24.575 

1.387 

17.71 

.01 

.27 

Control  Setting  Errors 

1.243 

0.610 

2.03 

.14 

— 

Sequence  Errors 

0.951 

0.494 

1.92 

.16 

— 

Success  Criteria 

0.681 

0.122 

5.58 

.01 

.09 
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General  Effects  of  Presentation  Method  Factor 


While  the  foregoing  results  reveal  some  reasonably  consistent  patterns  of  differences 
with  respect  to  task  performance  under  varying  instructional  presentation  methods,  they 
do  not  address  all  of  the  possible  contrasts.  Accordingly,  the  following  sections 
summarize  all  previously  unreported  comparisons  of  the  presentation  methods  in  which 
significant  differences  (i.e.,  £  <  .05  in  all  cases)  were  found  based  on  the  application  of 
Tukey's  post-hoc  analysis  procedure  to  the  ANOVA  results. 

Time-to-Completion 

For  this  measure,  results  showed  that,  on  the  average:  (1)  subjects  using  the  text- 
audio-graphics  method  completed  the  setup,  probe,  and  overall  tasks  faster  than  subjects 
who  used  the  text-only  and  audio-only  methods;  (2)  subjects  using  the  audio-graphics 
method  completed  the  setup  and  overall  tasks  faster  than  subjects  using  the  text-only, 
audio-only,  and  text-audio  methods;  and  (3)  subjects  using  the  text-graphics  method 
completed  the  setup  task  faster  than  subjects  using  methods  that  excluded  graphics,  and 
also  completed  tasks  on  an  overall  basis  faster  than  subjects  using  the  text-only  method. 

Repeats 

A  comparison  of  mean  values  for  this  measure  showed:  (1)  subjects  using  the  text- 
only  method  repeated  more  steps  on  the  separate  and  overall  tasks  than  subjects  using  the 
four  methods  that  included  auditory  instructions  (i.e.,  audio-only,  text-audio,  audio¬ 
graphics  and  text-audio-graphics);  (2)  subjects  using  the  text-only  method  repeated  more 
steps  than  subjects  using  the  text-graphics  method  on  the  setup,  probe,  and  overall  tasks; 
(3)  subjects  using  the  audio-only  and  text-audio  methods  repeated  more  steps  on  the  setup, 
probe,  and  overall  tasks  than  subjects  using  the  audio-graphics  method,  and,  similarly, 
repeated  more  steps  on  the  setup  and  overall  tasks  than  subjects  using  the  text-audio¬ 
graphics  method. 

Control  Setting  Errors 

This  analysis  showed  that  subjects  who  used  the  text-only  method  made  more  control 
setting  errors  on  the  setup  task  than  subjects  who  used  the  audio-graphics  method.  In 
addition,  it  was  found  that  subjects  using  the  text-graphics  method  made  more  control 
setting  errors  on  the  amplitude  task  than  subjects  using  the  text-only  method. 

Sequence  Errors 

For  this  measure  the  procedure  showed  that  subjects  using  the  three  methods  that 
excluded  graphics  made  more  sequence  errors  on  the  setup  task  than  subjects  using  the 
text-audio-graphics  method.  More  sequence  errors  were  also  committed  by  subjects  who 
used  the  text-only  method  on  the  setup  task  than  by  subjects  who  used  the  text-graphics 
and  audio-graphics  methods.  Further,  the  results  showed  that  subjects  who  used  the  text- 
only  and  audio-only  methods  made  more  sequence  errors  on  the  tasks  as  a  whole  than 
subjects  who  used  the  text-audio-graphics  method. 

Success  Criterion 


This  analysis  showed  that  a  larger  proportion  of  subjects  who  used  the  text-only 
method  failed  to  reach  the  criterion  for  successful  performance  on  the  setup  task  than 
subjects  who  used  the  text-audio-graphics  method. 
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Questionnaire  Data 


Differences  between  the  subject  groups  and  presentation  methods  were  thus  examin¬ 
ed  to  determine  their  effects  on  task  performance.  These  same  factors  were  included  in 
analyses  of  the  subjects'  mean  ratings  of  the  items  in  the  user  evaluation  questionnaire.7 
Specifically,  a  2  x  6  between-subjects  MANOVA,  the  results  of  which  are  summarized  in 
Table  6,  was  used  to  analyze  four  categories  of  questionnaire  responses.  The  first 
category  was  based  on  the  mean  of  ratings  assigned  to  items  1  through  25 — termed 
"Physical  Features";  the  second,  on  the  mean  of  ratings  assigned  to  items  26  through  36— 
termed  "User  Compatibility  Features";  the  third,  on  the  mean  of  ratings  assigned  to  items 
37  through  40— termed  "User  Acceptance";  and  the  fourth,  a  discrete  nominal  scale 
derived  from  positive,  negative,  or  no  responses  provided  in  the  last  section  of  the 
questionnaire— termed  "Comments." 


Table  6 

MANOVA  Results  for  User  Evaluation  Questionnaire  Data 


Part  I :  Multivariate  tests  with  20,  672  degrees  of  freedom  (  df ). 


Unit  of  Analysis 

TraceV 

F 

E< 

Group  x  Method 

0.126 

1.09 

.35 

Group  Effect 

0.046 

1.97 

.11 

Method  Effect 

0.231 

2.06 

.01 

Part  II :  Univariate  tests  with  5,  168  df.  a 

Effect  of  Presentation  Method 


Variable 

Numerator  MS 

Error  MS 

F 

R< 

Physical  Features 

0.980 

0.352 

2.78 

.05 

User  Compatibility 

1.627 

0.376 

4.33 

.01 

User  Acceptance 

1.187 

0.410 

2.90 

.05 

Comments 

0.307 

0.417 

0.73 

.60 

3. 

Univariate  results  are  not  listed  for  the  subject  group  by  presentation  method 
interaction  or  for  the  subject  group  variable  because  their  associated  multivariate 
F-ratios  were  nonsignificant  (see  Part  I  above). 


7  Responses  of  "Can't  Evaluate"  to  questionnaire  items  were  excluded  from  computa¬ 
tion  of  mean  rating  values  in  all  cases. 
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Inspection  of  Table  6,  Part  I,  shows  a  statistically  significant  F-ratio  for  the  main 
effect  of  presentation  method  in  relation  to  the  four  categories  of  questionnaire 
responses,  but  shows  nonsignificant  F-ratios  for  the  main  effect  of  subject  group  and  the 
group  by  method  interaction.  Based  on  these  findings,  it  is  appropriate  to  examine 
univariate  results  for  the  presentation  method  factor  to  identify  the  specific  categories 
of  questionnaire  responses  for  which  differences  were  found. 

The  latter  results,  which  are  summarized  in  Table  6,  Part  II,  show  significant  F-ratios 
for  all  but  the  "Comments"  variable.  Using  Tukey's  post-hoc  analysis  procedure  (with 
2  <  .05  in  all  cases),  it  was  found  that  the  mean  ratings  assigned  to  questionnaire  items 
dealing  with  physical  features  and  user  acceptance  considerations  were  higher  for 
subjects  who  used  the  text-audio-graphics  method  (4.40  and  4.44,  respectively)  than  for 
subjects  who  used  the  text-only  method  (3.94  and  3.90,  respectively).  In  regard  to  user 
compatibility,  the  Tukey  procedure  showed  higher  mean  ratings  for  subjects  who  used  the 
audio-graphics  and  text-audio-graphics  methods  (4.33  and  4.37,  respectively)  than  for 
subjects  who  used  the  text-only  method  (3.78);  and  further,  that  the  mean  rating  of 
subjects  who  used  the  text-audio-graphics  method  was  also  larger  than  the  3.88  mean 
rating  of  subjects  who  used  the  text-graphics  method.  All  other  pairings  did  not  differ 
significantly. 

Videotaped  Performances 


The  final  basis  for  assessing  the  two  independent  variables  used  in  this  study  involved 
reviewing  and  analyzing  task  performances  that  had  been  recorded  on  videotape  for  36 
subjects.  Particular  attention  was  given  in  these  analyses  to  identifying  any  general 
pattern (s)  of  differences  in  the  subjects'  approaches  to  performing  the  tasks  as  a  function 
of  the  grouping  and  presentation  method  factors.  Comparisons  of  recruit  and  BE/E 
subjects'  videotaped  performances  within  the  same  presentation  mode  showed  few,  if  any, 
discernible  differences  in  the  subjects'  behavior  when  receiving  the  information  presented 
in  the  task  steps,  and,  similarly,  when  actually  performing  the  tasks.  Discernible 
behavior  patterns  were  noted,  however,  when  performances  were  compared  among  the 
various  presentation  methods.  The  general  patterns  of  behavior  noted  in  this  context  are 
summarized  briefly  below. 

Subjects  assigned  to  the  text-only  and  text-graphics  methods  typically  read  each  task 
step  in  its  entirety  before  shifting  their  attention  to  the  oscilloscope  and  related  support 
equipment.  Once  this  shift  was  made,  subjects  then  proceeded  to  scan  the  equipment  to 
find  the  particular  control,  switch,  dial,  etc.,  described  in  the  task  step,  often  glancing 
back  to  the  text  on  the  computer  screen  for  confirmation  that  they  had,  indeed,  located 
the  correct  component.  In  many  cases,  however,  such  confirmation  was  not  possible 
because  the  text  had  already  disappeared  from  the  computer  screen.  As  a  result,  subjects 
using  these  two  methods  made  frequent  use  of  the  "REPEAT"  key,  particularly  during  the 
setup  and  probe  calibration  tasks,  but  less  so  during  the  amplitude  and  frequency 
measurement  tasks. 

A  similar  strategy  was  noted  among  subjects  assigned  to  the  text-audio,  audio¬ 
graphics,  and  text-audio-graphics  methods  during  their  performance  of  the  first  5  to  10 
steps  of  the  setup  task.  Shortly  thereafter  the  nature  of  the  subjects'  interaction  with  the 
instructional  material  changed.  They  began  directing  their  attention  to  the  oscilloscope 
first,  while  listening  to  the  instructions  presented  aurally,  occasionally  referring  back  to 
the  text  and/or  graphics  information  for  confirmation  of  specific  details  (e.g.,  discrete 
settings  for  the  Volts/CM  or  Time/CM  controls,  or  the  location  of  particular  controls). 
Moreover,  because  of  the  relatively  slow  rate  at  which  audio  instructions  were  presented, 
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subjects  who  used  these  methods  had  an  opportunity  to  refer  back  to  the  text  and/or 
graphics  information  before  it  disappeared  from  the  computer  screen.  This  pattern 
possibly  accounted  for  the  smaller  number  of  task  steps  repeated  by  subjects  who  used 
these  methods  relative  to  subjects  who  used  the  text-only  and  text-graphics  methods. 

The  performance  strategy  used  by  subjects  assigned  to  the  audio-only  method  was 
characterized  by  their  attending  directly  to  the  oscilloscope  from  virtually  the  very  first 
task  step  presented.  In  general,  subjects  who  used  this  method  limited  their  use  of  the 
repeat  key  to  the  more  lengthy  steps,  or  to  cases  where  they  could  not  readily  locate  the 
component(s)  specified  in  the  audio  instructions. 

With  respect  to  the  use  of  pictorial  information,  subjects  who  used  the  three  methods 
that  included  graphics  paid  close  attention  to  the  "flashing  arrows"  presented  during  the 
first  two  tasks,  but  much  less  so  on  subsequent  tasks.  Subjects  assigned  to  these  methods 
would,  on  occasion,  repeat  a  task  step  solely  for  the  purpose  of  studying  the  outline 
drawing,  but  this  activity  also  declined  steadily  over  time. 


DISCUSSION 

The  significant  differences  found  for  the  prior  training/experience  and  presentation 
method  factors  used  in  this  study  indicate  that  each  impacted  the  efficiency  and 
effectiveness  with  which  subjects  performed  the  four  oscilloscope  operator  tasks. 
Moreover,  the  interpretation  of  the  effects  attributable  to  these  factors,  discussed 
separately  in  the  following  sections,  is  direct  and  straightforward  because  none  of  the 
interactions  between  them  reached  conventional  levels  of  statistical  significance. 

Prior  Training/Experience 

On  an  overall  basis  the  results  showed  that  subjects  in  the  BE/E  group  completed  the 
tasks  faster,  repeated  fewer  task  steps,  made  fewer  control  setting  errors,  and  had  higher 
success  in  meeting  the  criteria  for  acceptable  performance  than  subjects  in  the  recruit 
group.  These  findings  not  only  support  the  first  experimental  hypothesis  but  also  are  in 
accord  with  some  of  the  findings  reported  in  studies  described  earlier  (Brown,  1964;  Goff 
et  al.,  1969),  and,  more  recently,  those  reported  by  Kieras,  Tibbits,  and  Bovair  (1984). 

It  is  interesting  to  note  that  the  four  types  of  performance  measures  on  which  the 
two  groups  differed  for  the  tasks  as  a  whole  were  not  always  those  that  differentiated  the 
groups  on  the  tasks  individually.  That  is,  only  the  time-to-completion  and  criterion 
measures  differed  significantly  between  groups  on  the  setup  and  frequency  measurement 
tasks,  respectively.  Several  factors  may  explain  these  findings.  With  respect  to  the  setup 
task,  all  subjects  in  both  groups  reported  that  they  had  no  prior  experience  in  operating 
the  particular  model  of  oscilloscope  used  in  this  study;  thus,  a  learning  effect  probably 
attenuated  group  differences  for  all  measures  except  time-to-completion.  The  fact  that 
the  BE/E  subjects  completed  this  task  faster  than  the  recruit  subjects  does  suggest, 
however,  that  their  familiarity  with  the  types  of  controls  and  actions  required  to  set  up  an 
oscilloscope  appears  to  have  generalized  to  the  equipment  used  in  this  study. 

A  plausible  explanation  for  the  lack  of  group  differences  on  four  of  the  measures  used 
to  assess  performance  on  the  frequency  task  is  that  the  subjects  were  required  to  use  a 
smaller  number  of  oscilloscope  controls.  That  is,  while  subjects  received  instructions  for 
manipulating  33  discrete  controls  in  the  setup  task,  only  5  controls  were  referenced  in  the 
instructions  for  the  frequency  task.  Moreover,  each  of  these  5  controls  had  been  used  at 
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least  once  in  performing  tasks  given  earlier  in  the  sequence.  It  is  reasonable  to  assume, 
therefore,  that  a  learning  effect  may  have  minimized  group  differences  on  this  task  for 
all  measures  except  the  success  criterion.  The  group  difference  on  the  latter  measure  can 
be  explained  largely  by  the  BE/E  subjects'  greater  familiarity  with  the  processes  involved 
in  measuring  and  computing  frequency  values. 

Graphics  Versus  Non-graphics  Methods 

Results  from  performance  comparisons  of  subjects  who  used  graphics  methods  with 
subjects  who  used  non-graphics  methods  were  generally  consistent  with  the  second 
experimental  hypothesis.  The  overall  measures  showed  that  subjects  assigned  to  the 
former  methods  completed  the  tasks  faster,  repeated  fewer  steps,  and  made  fewer 
sequence  errors  than  subjects  assigned  to  the  three  methods  that  excluded  graphics.  It  is 
important  to  note,  however,  that  the  combination  of  graphics  with  text  and/or  audio 
appears  to  have  had  its  most  pronounced  effect  on  the  subjects'  performance  early  in  the 
task  sequence.  That  is,  significant  differences  were  found  between  these  methods  on  all 
measures  for  the  setup  task;  whereas  only  the  time  and  repeat  measures  differed  between 
these  methods  on  the  probe  and  amplitude  tasks.  No  differences  were  found  between 
these  methods  on  the  frequency  task. 

A  possible  explanation  for  these  findings  in  the  case  of  the  probe  and  amplitude  tasks 
is  that  while  the  graphics  may  have  reduced  the  time  subjects  spent  in  searching  for 
particular  controls  or  in  repeating  task  steps,  they  appeared  to  have  had  little,  if  any, 
effect  on  the  accuracy  of  the  subjects'  performance  (i.e.,  as  reflected  by  control  setting 
error,  sequence  error,  and  criterion  measures).  The  lack  of  differences  between  these 
methods  when  comparing  the  subjects'  performance  on  the  frequency  task  can  be 
explained  for  much  the  same  reason  offered  previously.  That  is,  it  is  presumed  that  all 
subjects  had  become  sufficiently  familiar  with  the  types  of  controls  and  actions  used  in 
this  task  to  reduce  the  effectiveness  of  graphics.  This  effect  was  also  noted  in  the  review 
of  the  videotaped  performances  wherein  the  subjects'  reliance  on  graphics  information 
declined  steadily  over  time. 

These  findings  regarding  the  diminishing  effectiveness  of  graphics  to  procedural 
instructions  when  performing  similar  job  tasks  are  not  limited  to  the  present  study.  For 
example,  Smillie  and  Clelland  (1986)  reported  an  essentially  inverse  relationship  between 
the  preferences  of  Navy  fire  control  technicians  for  using  highly  pictorial  3PAs  and  the 
frequency  of  performing  job  tasks  supported  by  those  3PAs.  That  is,  the  technicians  they 
interviewed  expressed  a  preference  for  using  3PAs  to  support  the  performance  of  tasks 
that  had  long  intervals  of  time  between  them,  but  preferred  not  to  use  3PAs  on  tasks 
performed  routinely. 

The  lack  of  any  consistent  performance  differences  between  subjects  who  used  the 
graphics  and  non-graphics  methods  across  each  of  the  tasks  sampled  in  this  study  does 
conflict,  however,  with  the  findings  reported  by  Booher  (1975).  It  will  be  recalled  that 
subjects  in  that  study  who  received  instructions  through  predominantly  pictorial,  related- 
or  redundant-print  formats  performed  a  series  of  tasks  consistently  faster  and  with 
greater  accuracy  than  subjects  using  four  other  methods.  A  plausible  reason  for  this 
discrepancy  is  that  there  was  no  commonality  either  among  the  three  types  of  simulated 
job  tasks  that  Booher  sampled  or  among  the  eight  discrete  problems  included  under  each 
task  type.  Thus,  in  contrast  to  the  present  study,  Booher's  subjects  were  faced  with  an 
essentially  novel  situation  on  each  of  the  24  problems,  thereby  affording  little  opportunity 
for  familiarization  and/or  transfer  of  learning  to  occur  during  the  test  session. 
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Audio  Versus  Non-audio  Methods 


Among  the  more  interesting  findings  obtained  in  this  study  were  those  that  led  to 
rejection  of  the  third  and  fourth  experimental  hypotheses.  Contrary  to  the  third 
hypothesis,  results  showed  that  subjects  assigned  to  the  text-only  method  repeated  more 
steps  on  the  separate  and  overall  tasks  and  were  less  successful  in  meeting  the  criterion 
for  acceptable  performance  on  the  setup  task  than  subjects  who  used  the  audio-only  and 
text-audio  methods.  Similar  contradictory  findings  were  obtained  for  the  fourth  hypothe¬ 
sis;  subjects  assigned  to  the  text-graphics  method  repeated  more  steps  on  the  separate 
and  overall  tasks,  made  more  control  setting  errors  on  the  amplitude  task,  and  had  higher 
failure  rates  on  the  probe  and  overall  tasks  than  subjects  who  used  the  audio-graphics  and 
text-audio-graphics  methods. 

Taken  together,  these  findings  suggest  that  the  task  performance  of  subjects  assigned 
to  the  methods  that  included  audio  instructions  was  generally  more  efficient  (i.e.,  fewer 
repeated  steps),  and  in  some  cases  more  effective  (i.e.,  fewer  errors  and  lower  failure 
rates)  than  that  of  subjects  assigned  to  methods  that  excluded  audio  instructions.  Once 
again,  these  findings  are  consistent  with  those  from  earlier  studies. 

Results  from  applied  research  by  Henneman  (1952)  and  his  associates  (e.g,  Henneman, 
Lewis  5c  Matthews,  1953;  Henneman  <5c  Matthews,  1954;  Holland  <5c  Lee,  1954),  showed,  for 
example,  that  when  subjects  were  engaged  in  highly  attention-demanding  tasks,  messages 
presented  aurally  were  more  accurately  received  and  interfered  less  with  the  performance 
of  simultaneous  motor  tasks  than  identical  messages  presented  visually.  Evidence  for  the 
superiority  of  aural  over  visual  messages  when  subjects  performed  simultaneous  motor  or 
information  processing  tasks  was  also  obtained  in  applied  research  conducted  by  Chapanis 
(1973)  and  his  associates  (e.g.,  Chapanis,  Ochsman,  Parrish,  <5c  Weeks,  1972;  Ochsman  & 
Chapanis,  1974).  A  relatively  straightforward  explanation  for  these  findings  was  offered 
in  the  latter  study.  It  stated  that: 

Subjects  in  the  hard-copy  modes  tend  to  concentrate  more  on  their 
communication  task,  i.e.,  on  sending  or  receiving.  Subjects  who 
communicate  by  speaking  find  it  easier  to  conduct  other  activities 
simultaneously  regardless  of  whether  they  are  speaking  or  listening  to 
spoken  messages.  (Ochsman  &  Chapanis,  1974,  p.  614) 

Findings  from  more  basic  research  studies  have  also  repeatedly  shown  that  memory 
traces  laid  down  during  the  presentation  of  visual  stimuli  are  less  persistent  and  more 
vulnerable  to  interference  than  traces  from  the  same  information  presented  aurally  (e.g., 
Broadbent,  1956;  Craik,  1969;  Laughery  &  Fell,  1969;  Margrain,  1967;  Mowbray,  1952; 
Murdock,  1968;  Neisser,  1967;  Sperling,  1967;  Wickelgren,  1965).  Thus,  in  the  present 
study,  the  consistently  higher  number  of  steps  repeated  by  subjects  who  used  the  text-only 
and  text-graphics  methods  compared  with  subjects  whose  methods  included  audio  instruc¬ 
tions  would  appear  to  be  in  general  agreement  with  the  findings  from  the  studies  cited 
above. 

Another  possible  explanation  for  these  findings  was  noted  after  review  of  the 
videotaped  performances  of  subjects  assigned  to  the  text-only  and  text-graphics  methods. 
Specifically,  it  was  noted  that  the  text  often  disappeared  from  the  computer  display 
before  the  subjects  could  go  back  to  re-read  it.  As  a  result,  the  instructions  presented  in 
these  methods  lacked  referability;  Goldstein  (1940)  reported  that  referability  was  a 
critical  factor  determining  the  relative  effectiveness  of  text  and  audio  presentations. 
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The  lack  of  referability  of  information  presented  in  the  text-only  and  text-graphics 
methods  notwithstanding,  it  is  also  possible  that  the  generally  poorer  performance  of 
subjects  who  used  the  non-audio  methods  might  be  attributable  to  the  physical  distance 
that  existed  between  the  source  of  information  (i.e.,  the  microcomputer)  and  the  work 
(i.e.,  the  oscilloscope).  It  would  be  of  interest,  therefore,  to  determine  if  a  similar 
pattern  of  performance  differences  would  result  between  subjects  who  used  the  audio  and 
non-audio  methods  when  task  instructions  for  the  latter  were  presented  in  closer 
proximity  to  the  equipment  used  to  perform  the  job  tasks. 

User  Attitudes 


To  a  certain  extent,  the  foregoing  performance  differences  between  subjects  who 
used  methods  that  combined  audio  instructions  with  graphics  and  those  who  did  not  were 
also  reflected  in  the  subjects'  evaluations  of  those  methods.  Specifically,  analyses  of  the 
user  evaluation  questionnaire  data  showed  that  subjects  assigned  to  the  audio-graphics  and 
text-audio-graphics  methods  provided  significantly  higher  (i.e.,  more  favorable)  ratings  to 
items  dealing  with  user  compatibility  than  subjects  assigned  to  the  text-only  and  text- 
graphics  methods.  These  data  also  showed  that  subjects  who  used  the  text-only  method 
provided  significantly  lower  ratings  than  did  subjects  who  used  the  text-audio-graphics 
method  on  items  concerned  with  the  physical  features  of  the  hardware  used  in  performing 
the  tasks,  as  well  as  on  those  dealing  with  user  acceptance. 

No  significant  differences  could  be  attributed  to  the  various  presentation  methods 
with  regard  to  open-ended  comments  provided  by  subjects  in  the  final  section  of  the 
questionnaire.  It  is  interesting  to  note,  however,  that  the  majority  of  negative  comments 
made  by  subjects  who  used  the  text-only  and  text-graphics  methods  indicated  that  more 
time  should  have  been  allowed  for  presenting  the  task  steps.  Subjects  assigned  to  methods 
that  included  audio  instructions  made  no  such  comments  about  the  rate  at  which  steps 
were  presented  aurally.  It  would  appear,  therefore,  that  the  lack  of  referability  in  the 
textual  instructions  contributed  to  many  of  the  negative  comments  made  by  subjects  who 
used  the  text-only  and  text-graphics  methods. 


SUMMARY  AND  CONCLUSIONS 

This  study  was  designed  to  assess  systematically  the  effects  of  prior  task  training  or 
experience,  and  alternative  methods  for  presenting  instructional  materials  on  job  task 
performance.  Results  showed  significant  main  effects  for  each  of  these  factors 
separately;  however,  none  of  the  two-way  interaction  effects  between  these  factors 
reached  conventional  levels  of  statistical  significance.  The  conclusions  and  implications 
derived  from  this  study  are  discussed  in  more  detail  below. 

It  is  interesting  to  note  that  prior  training  and  experience  in  operating  dual-trace 
oscilloscopes  had  their  most  pronounced  effects  on  the  outcome  measures  obtained  for  the 
probe  and  amplitude  tasks;  whereas  only  the  time-to-completion  and  success  criterion 
measures  differed  between  groups  on  the  setup  and  frequency  tasks,  respectively.  One 
implication  derived  from  these  findings  is  that  learning  likely  played  an  important  role  in 
moderating  differences  between  groups  on  the  initial  and  final  tasks  performed  in  the 
sequence.  Such  learning  effects  represent  an  additional  source  of  variance  that  must  be 
considered,  and  preferably  factored-out  in  future  research  efforts  concerned  with 
assessing  job  performance. 
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Results  from  the  test  of  the  second  experimental  hypothesis  are  generally  in  accord 
with  the  predictions  of  Paivio's  (1971,  1978)  dual-coding  theory.  By  conveying  information 
through  both  verbal  and  image  codes,  subjects  were  provided  with  complementary 
processing  systems,  thereby  enabling  them  to  alternate  between  the  two  to  obtain  the 
information  needed  to  perform  the  job  tasks.  Thus,  each  system  could  serve  as  an 
alternative  representation  of  the  information  should  there  be  any  ambiguity  (i.e.,  the 
graphics  appeared  to  serve  as  a  reference  point  for  the  instructions  presented  in  text 
and/or  audio  form,  while  the  latter  appeared  to  serve  as  a  focusing  agent  for  the 
information  presented  in  graphics  form). 

It  is  important  to  note,  however,  that  the  combination  of  verbal  and  image  codes  had 
greatest  effect  on  the  performance  of  tasks  administered  early  in  the  sequence,  when 
subjects  were  learning  the  location  and  function  of  particular  oscilloscope  controls 
specified  in  the  instructions.  One  of  the  implications  of  these  findings  is  that,  once 
learning  has  occurred,  the  effect  of  combined  verbal  and  image  code  presentations  will  no 
longer  be  observed.  This  suggests  a  refinement  to  the  dual-coding  theory  in  that  the 
maximum  effect  of  combined  presentations  will  be  during  the  initial  learning  of  task 
behaviors  rather  than  during  continuous  performance  of  those  behaviors. 

In  the  present  study,  subjects  who  used  methods  that  included  audio  instructions 
generally  performed  better.  It  may,  under  most  conditions,  be  most  efficient  to  present 
job  task  instructions  via  audio.  This  conclusion  is  supported  by  both  basic  and  applied 
research  studies  that  have  repeatedly  demonstrated  that  information  presented  aurally 
persists  longer  in  short-term  memory  and  is  less  vulnerable  to  interference  than  visually 
presented  information.  It  is  worth  mentioning  that  these  findings  conflict  with  the 
empirical  evidence  interpreted  in  support  of  the  information  processing  and  communica¬ 
tion  theories  discussed  in  the  second  section  of  this  report  (e.g.,  Hsia,  1977;  Paivio,  1978; 
Van  Mondfrans  &  Travers,  1964).  Probably  the  most  reasonable  explanation  for  the 
apparent  discrepancy  between  these  findings  is  that  the  latter  studies  placed  a  premium 
on  the  subjects'  ability  to  recall  or  recognize  the  stimulus  material  presented;  whereas  in 
the  present  study  subjects  were  required  to  perform  some  action(s)  based  on  the  stimulus 
material  presented. 

For  the  types  of  job  performance  tasks  and  experience  levels  sampled,  results  showed 
that  the  performance  of  subjects  who  used  the  audio-graphics  and  text-audio-graphics 
methods  was  generally  more  efficient  and  effective  than  that  of  subjects  who  used  the 
other  presentation  conditions.  Additionally,  the  responses  to  questionnaire  items  by 
subjects  who  used  these  methods  were  generally  more  favorable  with  respect  to  the 
physical  features  of  the  hardware,  user  compatibility,  and  user  acceptance. 


RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

The  empirical  evidence  suggests  the  presence  of  learning  effects  across  the  series  of 
consecutively  administered  job  tasks.  These  effects  tended  to  minimize  differences 
between  groups  for  both  the  prior  training  and  experience  variable  and  the  graphics  versus 
non-graphics  comparisons.  Such  findings  suggest  that  further  research  in  this  area  is 
justified.  Further  study  is  also  warranted  to  determine  whether  the  generally  superior 
performance  of  subjects  who  used  the  audio-graphics  and  text-audio-graphics  methods 
would  generalize  to  other  job  situations  in  which  the  supporting  information  was  not  in  the 
form  of  highly  proceduralized,  step-by-step  instructions.  An  example  of  the  foregoing  is 
troubleshooting  activities  that  often  involve  circuit  or  signal  tracing  using  schematic 

diagrams,  logic  trees,  or  "narrative"  descriptions  of  circuit  functions  and  their  interrela¬ 
tionships. 
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Specific  research  recommendations  for  improving  the  design  and  methodology  of 
related  future  studies  are  summarized  below: 

1.  To  control  for  learning  effects  when  performing  job  tasks  with  unfamiliar 
equipment,  all  subjects  should  demonstrate  mastery  (or  the  lack  of  it)  with  regard  to  the 
location  and  function  of  controls  on  the  equipment  to  be  used  in  the  test  situation. 

2.  The  job  tasks  selected  for  testing  should  be  relatively  independent  of  one 
another,  and  of  comparable  difficulty  level,  to  enable  the  use  of  a  counterbalanced  task 
presentation  order. 

3.  The  task  instructions  in  the  text-only  and  text-graphics  methods  should  be 
presented  in  closer  proximity  to  the  equipment  used  to  perform  the  job  tasks.  This  change 
might  produce  results  that  are  more  consistent  with  the  third  and  fourth  hypotheses 
tested  in  this  study. 

4.  Discrete  measures  should  be  obtained  on  each  procedural  step  to  enable  a 
comparative  assessment  of  differences  attributable  to  short  versus  lengthy  procedural 
instructions. 
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APPENDIX  A 

BACKGROUND  INFORMATION  FORM 


A-0 


AN/USM-281  OSCILLOSCOPE  OPERATOR  TEST: 

BACKGROUND  INFORMATION  FORM 

SUBJECT  NUMBER: _  CLASSIFICATION:  RECRUIT _  BE/E  STUDENT _ 

TEST  DATE:  _  PRESENTATION  MODE:  _ 

1.  How  old  are  you?  _ _ _ 

2.  What  is  the  highest  level  of  education  you  completed  before 
entering  the  Navy? 


3.  Is  English  your  native  language?  Yes _  No _ .  If  no, 

specify  native  language. 


4.  Do  you  have  any  vision  and/or  hearing  problems  that  you  are 

aware  of  at  the  present  time?  No _  Yes _ .  If  yes,  specify 

type  and  extent  of  problem(s). 


5  .  How  long  have  you  been  in  the  Navy?_ _ 

6.  What  Navy  rating  do  you  intend  to  have  at  the  end  of  your 
current  enlistment  period  (e.g.,  electronics  technician,  fire 
control  technician,  sonar  technician)? 


7 .  Did  you  have  any  prior  military  service  before  entering  the 

Navy?  No  _  Yes  _ .  (If  yes,  specify  which  service  and  any 

occupational  specialty (ies)  held.) 
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8.  Before  entering  the  Navy,  did  you  have  any  formal  schooling  or 
training  in  the  areas  of  electricity  or  electronics?  No 

Yes _ •  (If  yes,  specify  the  nature  and  type  of 

schooling/training  received;  e.g. ,  high  school  shop  class(es), 
vocational/trade  school,  factory-sponsored  training,  etc.) 


9.  Before  entering  the  Navy,  did  you  have  any  work  experience  in 

the  area  of  electricity  or  electronics?  No  _  Yes  .  (If 

yes,  specify  the  nature  and  type  of  work  experience  received;  e.g.  , 
part-time  job  in  TV  repair  shop,  apprentice  electrician  or  helper, 
etc.  ) 


10.  Have  you  ever  operated  an  oscilloscope  before?  No 

Yes _ *  (If  yes,  specify  when,  where,  and  how  much  oscilloscope 

operation  experience  was  accumulated  and  the  type(s)  of 
oscilloscopes  used.) 


APPENDIX  B 

STANDARDIZED  CONTROL  SETTINGS  FOR  AN/USM-281  OSCILLOSCOPE 
AND  RELATED  SUPPORT  EQUIPMENT 


B-0 


AN/USM-281  OSCILLOSCOPE  OPERATOR  TEST 


SPECIFICATION  SHEET  #2 


The  following  describes  the  procedure  for  undercompensating  the 
divider  probe  and  lists  required  control  settings  and  adjustments  on 
the  USM-281  oscilloscope  before  administering  the  PROBE  CALIBRATION 
TASK. 


PART  I.  Procedure  for  Undercompensating  the  Divider  Probe 
Position  controls  and  switches  on  USM-281  oscilloscope  as  follows: 

(a)  Set  CHANNEL  A  INPUT  COUPLING  switch  to  AC. 

(b)  Rotate  CHANNEL  A  VOLT/CM  control  to  .5  Volt/CM. 

(c)  Rotate  MAIN  TIME/CM  control  to  .5  MSEC. 


Perform  the  following  adjustments  to  the  divider  probe: 

(a)  Remove  probe  body  and  tip  assembly  and  attach  banana  plug 
adapter  to  threaded  end  of  probe  body. 

(b)  Loosen  locking  sleeve  on  divider  probe  by  turning  it 
counterclockwise  one  full  turn. 

(c)  Connect  BNC  coupling  on  divider  probe  to  CHANNEL  A  INPUT 
connector  post. 

(d)  Attach  probe  ground  lead  (alligator  clip)  to  CALIBRATOR 
GND  jack;  insert  banana  plug  in  CALIBRATOR  10V  jack. 

(e)  Adjust  FOCUS  and  CHANNEL  A  POSITION  controls  as 
necessary  to  present  a  "sharp"  image,  centered  on  the 
the  oscilloscope  display  screen. 

(f)  Rotate  probe  body  clockwise  until  leading  edge  of  upper 
square  waveform  and  leading  edge  of  lower  square  waveform 
"curve"  toward  the  horizontal  graticule  line  at  the  center 
of  the  oscilloscope  display. 

(g)  Tighten  locking  sleeve  on  divider  probe  by  turning  it 
clockwise  until  finger-tight;  ensure  that  undercompensated 
waveform  is  still  present  on  oscilloscope  display  after 
after  locking  sleeve  has  been  tightened. 
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SPECIFICATION  SHEET  #2  (Cont'd) 


(h)  Remove  BNC  coupling  on  divider  probe  from  CHANNEL  A 
INPUT  connector  post,  probe  ground  from  CALIBRATOR  GND 
jack,  and  banana  plug  from  CALIBRATOR  10V  jack. 

(i)  Remove  banana  plug  adapter  from  end  of  probe  body 
and  reinstall  "original"  probe  body  and  tip  assembly. 


PART  II.  Pre-Test  Settings  for  USM-281  Oscilloscope 


When  the  divider  probe  has  been  undercompensated,  ensure  that 
oscilloscope  switch  and  control  settings  are  positioned  as  follows: 

OSCILLOSCOPE  CORRESPONDING 

CONTROLS /SWITCHES  PRE-TEST  SETTINGS  STEP  NUMBER 


INTENSITY 

Center  of  rotation 

1 

FOCUS 

Center  of  rotation 

2 

POWER 

OFF 

3 

HORIZONTAL  POSITION 

Center  of  rotation 

4 

HORIZONTAL  MAGNIFIER 

XI 

5 

HORIZONTAL  DISPLAY 

INT 

6 

HORIZONTAL  AC -DC  SWITCH 

AC 

7 

MAIN  VERNIER 

Clockwise  to  locked  position 

8 

DELAYED  VERNIER 

Clockwise  to  locked  position 

9 

SWEEP  DISPLAY 

Fully  clockwise 

10 

MAIN  TIME/CM 

1  USEC 

11 

DELAYED  TIME /CM 

OFF 

12 

MAIN  TRIGGER  LEVEL 

Mid-position 

13 

DELAYED  TRIGGER  LEVEL 

Mid-position 

14 

SWEEP  MODE 

AUTO 

15 

MAIN  TRIGGER  SOURCE 

INT 

16 

DELAYED  TRIGGER  SOURCE 

AUTO 

17 

MAIN  SLOPE 

Plus  (4-)  sign 

18 

DELAYED  SLOPE 

Plus  (+)  sign 

19 

MAIN  TRIGGER  COUPLING 

AC 

20 

DELAYED  TRIGGER  COUPLING 

AC 

21 

CHANNEL  A  POSITION 

Center  of  rotation 

22 

CHANNEL  B  POSITION 

Center  of  rotation 

23 

VERTICAL  DISPLAY 

"A” 

24 

CHANNEL  A  POLAR TTY 

Plus  (4-)  sign  UP 

25 

CHANNEL  B  POLARITY 

Plus  (4-)  sign  UP 

26 

CHANNEL  A  VERNIER 

Clockwise  to  locked  position 

27 

CHANNEL  B  VERNIER 

Clockwise  to  locked  position 

28 

CHANNEL  A  VOLT/CM 

5  Volt/CM 

29 

CHANNEL  B  VOLT/CM 

5  Volt/CM 

30 

VERTICAL  MAGNIFIER 

XI 

31 

CHANNEL  A  INPUT  COUPLING 

GND 

32 

CHANNEL  B  INPUT  COUPLING 

AC 

33 
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APPENDIX  C 

EXAMPLE  OF  A  PERFORMANCE  OBSERVATION  FORM 
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PERFORMANCE  OBSERVATION  FORM  #1 


PRELIMINARY  SET-UP  AND  ADJUSTMENT  TASK 


Examinee  Number:  _  Classification:  Recruit  _  BE/E 

Test  Date:  _  Presentation  Mode: 


Instructions :  Score  one  point  for  each  step  performed  correctly  and  one  point  for 
correct  sequencing  of  each  step.  Score  0  for  an  incorrect  performance  or 
sequencing  of  steps;  indicate  specific  error (s)  made  under  the  column  headed 
"Comments . " 


I 

Step 

i 

Control -Operation 

1 

1 

1 

Correct 

Setting 

i 

l 

Correct 

Sequence 

i 

l 

I 

Comments  | 

1 

INTENSITY- -12  o'clock 

1 

1 

l 

I 

1 

2 

FOCUS- -12  o'clock 

1 

1 

1 

1 

l 

l 

3 

|  POWER- -OFF 

1 

1 

1 

4 

HORIZONTAL  P0S.--12  o'clock 

1 

1 

1 

l 

5 

i  HORIZONTAL  MAG.- -XI 

1 

1 

l 

I 

l 

1 

6 

|  HORIZONTAL  DI SPLAY- -INT 

1 

1 

I 

1 

7 

j  HORIZONTAL  AC -DC- -AC 

1 

1 

I 

l 

1 

8 

|  MAIN  VERNIER- -fully  CW 

l 

1 

9 

|  DELAYED  VERNIER--fully  CW 

1 

l 

l 

l 

1 

10 

|  SWEEP  DISPLAY- -fully  CCW 

1 

1 

1 

1 

l 

11 

|  MAIN  TIME/CM--1  USEC 

1 

1 

1 

I 

1 

1 

12 

|  DELAYED  TIME/CM- -OFF 

1 

1 

I 

1 

1 

13 

!  MAIN  TRIG.  LEVEL- -mid  pos . 

1 

1 

I 

l 

l 

14 

i  DEL.  TRIG.  LEVEL- -mid  pos. 

1 

1 

l 

l 

l 

15 

j  SWEEP  MODE --AUTO 

1 

l 

l 

16 

|  MAIN  TRIGGER  SOURCE- -INT 

1 

1 

l 

l 

l 

l 

17 

|  DELAYED  TRIG.  SOURCE --AUTO 

1 

1 

I 

l 

l 

l 

C-l 


r 

T 

Correct 

i 

l 

Correct 

u — 

1 

Step 

Control -Operation 

1 

1 

Setting 

1 

Sequence 

l 

! 

Comments 

18 

|  MAIN  SLOPE-- (+)  sign 

1 

1 

i 

1 

1 

19 

DELAYED  SLOPE-- (+)  sign 

1 

1 

l 

l 

1 

20 

|  MAIN  TRIGGER  COUPLING- -AC 

1 

1 

i 

1 

1 

21 

|  DELAYED  TRIG.  COUPLING- -AC 

1 

1 

1 

22 

CHANNEL  A  POS.--12  o'clock 

1 

1 

1 

1 

23 

CHANNEL  B  POS.--12  o'clock 

1 

1 

1 

1 

24 

VERTICAL  DISPLAY--"A" 

1 

1 

1 

25 

|  CHANNEL  A  POLARITY-- (+)  UP 

1 

1 

l 

1 

1 

26 

|  CHANNEL  B  POLARITY-- (+)  UP 

1 

1 

i 

i 

1 

1 

27 

CHANNEL  A  VERNIER- -fully  CW 

1 

1 

1 

1 

1 

28 

|  CHANNEL  B  VERNIER--fully  CW 

1 

1 

l 

i 

1 

1 

29 

|  CHANNEL  A  VOLT/CM- -5  volt/cm | 

l_  .  1 

1 

1 

1 

1 

30 

|  CHANNEL  B  VOLT/CM--5  volt/cm | 

1  1 

1 

1 

1 

31 

1  VERTICAL  MAGNIFIER- -XI 

1 

1 

i 

i 

1 

32 

CHAN.  A  INPUT  COUPLING- -GND 

1 

1 

.1 

1 

1 

33 

CHAN.  B  INPUT  COUPLING- -AC 

1 

1 

i 

1 

1 
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APPENDIX  D 

EXAMINEE  WORKSHEET  FORM 


D-0 


EXAMINEE  WORKSHEET 


Examinee  Number: _  Classification:  Recruit _  BE/E _ 

Test  Date: _ _  Presentation  Mode: _ 

AMPLITUDE  MEASUREMENT  CALCULATIONS 

Step  66.  Number  of  centimeters  counted  =  _ 

Step  67.  CHANNEL  A  VOLTS/CM  setting  x  10  =  _ 

Step  68.  Result  Step  66  x  Result  Step  67  =  _ 

Step  69.  Result  Step  68 

VERTICAL  MAGNIFIER  setting 

FREQUENCY  MEASUREMENT  CALCULATIONS 

Step  81.  Number  of  centimeters  counted  =  _ 

Step  82.  Result  Step  81  x  MAIN  TIME/CM  setting  =  _ 

Step  83.  Result  Step  82  x  0.000001  =  _ 

Step  84.  Result  Step  83 

HORIZONTAL  MAGNIFIER  setting 

Step  85.  1 

Result  Step  84 
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APPENDIX  E 

USER  EVALUATION  QUESTIONNAIRE  FORM 
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USER  EVALUATION  QUESTIONNAIRE 


Examinee  Number:  _  Classification:  Recruit  _  BE/E 

Test  Date:  Presentation  Mode: 


Directions : 


You  are  asked  to  evaluate  the  items  on  this  questionnaire  using  the  5 -point 
scale  appearing  to  the  right  of  each  item.  Rate  each  item  by  placing  an  "X"  in 
the  appropriate  column.  Because  the  same  questionnaire  will  be  given  to  all 
examinees,  there  may  be  some  items  you  cannot  evaluate  due  to  the  particular 
method  and  equipment  you  used  for  receiving  the  performance  steps.  In  those 
cases,  please  place  an  "X"  in  the  column  headed:  "Can't  Evaluate." 

SECTION  1:  PHYSICAL  FEATURES 

Items  1  through  25  concern  the  following  physical  features  of  the 
equipments  and  methods  used  for  performing  the  oscilloscope  operator  tasks: 

--The  layout  of  equipment  on  the  workbench, 

--The  two  push-button  "Command"  keys  located  on  the  computer 
keyboard, 

--The  computer  display  screen  and  audio  equipment  used  to  present 
the  performance  steps . 


SCALE  VALUES 


ITEMS 


1.  Physical  distance  between  you  and 
the  equipment (s)  used  to  present 
the  performance  steps  (e.g.,  the 
computer  display,  keyboard,  and 
loudspeaker) . 


2.  Physical  distance  between  the 

equipment (s)  used  to  present  the 
performance  steps  and  the  equip¬ 
ment  required  to  perform  the 
tasks  (i.e.,  the  oscilloscope, 
divider  probe’,  and  black  box)  . 
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SCALE  VALUES 


ITEMS 
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3.  Physical  placement  of  equipments 
on  the  workbench.  (I.E.,  Was  the 
equipment  arranged  in  a  useable 
and  functional  order?) 


4.  Impression  of  overall  layout  of 
equipments  on  the  workbench. 


5.  Adequacy  of  identification  labels 
for  switches  and  controls  on  the 
equipment  required  to  perform  the 
tasks . 


6.  Location  of  ADVANCE  and  REPEAT 
keys  on  the  computer  keyboard. 


7.  Spacing  of  ADVANCE  and  REPEAT 
keys  . 


8.  Ease  of  operating  ADVANCE  and 
REPEAT  keys. 


9.  Indication (s )  that  ADVANCE  and 
REPEAT  keys  had  been  activated. 


10.  Reliability  of  the  ADVANCE  and 
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12.  Glare  resistance  of  the  computer 
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13.  Legibility  of  printed  performance 
steps.  (I.E.,  Were  letters,  words, 
and  numbers  easy  to  read?) 


14.  Format/arrangement  of  printed 
steps  (i.e.,  lack  of  clutter 
and  crowding) . 


15.  Clearness  of  performance  steps 
read  aloud  to  you  (i.e.,  not 
garbeled) . 


16.  Volume  level  of  the  steps  read 
aloud  to  you. 


17.  Size  of  outline  drawings  provided 
for  oscilloscope,  divider  probe, 
and  black  box. 


18.  Interpretability  of  outline  draw¬ 
ings  (i.e.,  lack  of  clutter  and 
crowding) . 


19.  Format/arrangement  of  outline 
drawings.  (I.E.,  Were  they  pre¬ 
sented  in  a  logical  and  under¬ 
standable  way?) 


20.  Resolution/clarity  of  outline 
drawings . 


21.  Brightness  of  outline  drawings . 


22.  Contrast  between  outline  drawings 
and  background  on  the  computer 
display  screen. 
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23.  Adequacy  of  detail  provided  on 
outline  drawings. 


24.  Adequacy  of  outline  drawings  for 
supplementing  printed  and/or 
spoken  performance  steps . 


25.  Adequacy  of  "f lashing  arrowM  to 
highlight  switches  and  controls 
on  the  outline  drawings. 
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SECTION  2:  PRESENTATION  METHODS 


Items  26  through  40  concern  the  following  aspects  of  the  methods  used  for 
presenting  the  oscilloscope  operator  task  instructions: 

--Organization,  sequencing,  and  pacing  of  the  performance  steps, 

--Clarity/completeness  of  information  contained  in  the 
performance  steps , 

--Ease  of  performing  actions  specified  in  the  performance  steps, 

--Overall  impression  of  the  method  you  used  for  receiving  the 
performance  steps. 


To  avoid  repetition  in  the  wording  of  the  items  contained  in  this 
section,  please  begin  each  with  the  phrase: 


BASED  ON  THE  METHOD  YOU  USED  FOR  RECEIVING  THE 
PERFORMANCE  STEPS,  HOW  WOULD  YOU  RATE  IT  IN  TERMS  OF... 


SCALE  VALUES 
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26.  Completeness  of  information 
provided . 


27.  Ordering  of  steps  in  the  task 
sequences 


28.  Clarity  of  the  wording  in  the 
performance  steps.  (I.E.,  Did 
each  step  describe  exactly  what 
action(s)  to  perform?) 


29.  Conciseness  of  the  wording  in 

the  performance  steps.  (I.E.,  Did 
each  step  describe  the  action(s) 
to  perform  using  a  minimum  number 
of  words?) 
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BASED  ON  THE  METHOD  YOU  USED  FOR  RECEIVING 
PERFORMANCE  STEPS,  HOW  WOULD  YOU  RATE  IT  IN  TERMS  OF. 


ITEMS 


30 .  Rate  at  which  the  performance 
steps  were  presented. 
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31.  Ease  of  using  the  information  to 

locate  controls  and  switches  on  the 
oscilloscope,  divider  probe,  and 
black  box  quickly  and  accurately. 
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32.  Ease  of  using  the  information  to 
perform  the  action(s)  specified 
in  the  steps . 
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33.  Minimizing  physical  fatigue  (e.g. 
tired  eyes,  backache,  etc.). 
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34.  Minimizing  mental  fatigue  (e.g., 
being  overloaded  wTith  information, 
confused,  frustrated,  etc.). 


35.  Minimizing  the  requirement  to 
repeat  the  same  performance 
step(s) . 


36.  Minimizing  the  time  and  effort 
required  to  perform  the  oscillo¬ 
scope  operator  tasks. 


37.  Your  level  of  confidence  in 
performing  the  oscilloscope 
operator  tasks. 


BASED  ON  THE  METHOD  YOU  USED  FOR  RECEIVING  THE  PERFORMANCE 
STEPS,  HOW  WOULD  YOU  RATE  IT  IN  TERMS  OF.  . . 


ITEMS 


38.  Your  level  of  confidence  in  using 
.this  method  to  perform  other  types 
of  job  tasks. 


39.  Its  adequacy  and  acceptability 

for  presenting  information  needed 
to  perform  job  tasks. 


40.  Its  overall  adequacy  and  accept¬ 
ability  . 
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SECTION  3:  GENERAL  REACTIONS  AND  COMMENTS 


This  section  provides  spaces  for  making  any  comments,  complaints, 
suggestions,  etc.  you  have  regarding  the  method  you  used  to  receive  instructions 
for  the  oscilloscope  operator  tasks. 


This  concludes  the  user  evaluation  questionnaire.  Your  assistance  in  providing 
this  information  is  appreciated. 
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